Engineering

Engineering

Come for answers, stay for best practices. All we're missing is you.

 View Only

  • 1.  Seeking Expertise: Best Practices for Real-Time Streaming Architecture

    Posted 07/30/26 05:44 AM
    Hello fellow engineers,

    In a recent discussion thread about optimizing data pipelines, we've hit a roadblock regarding the most efficient way to handle real-time streaming data at scale. While we've considered several architectures, we haven't yet settled on a solution that meets both performance and cost-effectiveness.

    Could any of you with experience in high-throughput streaming systems provide guidance or share best practices? Specifically, we'd appreciate insights on:
    • Preferred technologies or frameworks (e.g., Kafka, Kinesis, Spark Structured Streaming)
    • Design patterns for fault tolerance and scalability
    • Monitoring and management tools you find indispensable
    Any pointers, case studies, or lessons learned would be greatly appreciated. Thank you in advance for your assistance!


  • 2.  RE: Seeking Expertise: Best Practices for Real-Time Streaming Architecture

    Posted 08/02/26 03:37 AM

    In order to do this, we really need to know more about what you are doing and what the data looks like - I am a big Kafka fan but I am primarily working with IoT sensor data. 

    The data:

     - what is the 'shape' of your data? Size, format, content type, encryption

     - what volume are you processing? steady state and/or burst? continuous?

     - how many sources does it come from?

    The processing:

     - what do you need to do with the data? Filtering/transforming/aggregating/storing

     - what's the ultimate destination? Is the data kept 'as is' or is it used to create some other structures?



    ------------------------------
    Davyd Norris
    Principal Consultant | Founder
    Olive Grove IT Pty Ltd
    Camberwell
    +61 414 299 619
    ------------------------------



  • 3.  RE: Seeking Expertise: Best Practices for Real-Time Streaming Architecture

    Posted 08/10/26 02:27 AM

    Interesting problem. When you mention "real-time streaming data at scale," are you primarily dealing with application/business events, CDC, IoT/telemetry, or another type of data?

    A few things would help understand the architecture:

    1. What are the current event producers and consumers?
    2. Are you currently using Kafka/Kinesis/Spark in production, or evaluating them?
    3. Is the main challenge throughput/latency, reliability, operational complexity, or infrastructure cost?
    4. Do you need real-time transformation/enrichment/aggregation, or primarily reliable event distribution?
    5. Are there existing enterprise messaging platforms such as IBM MQ that need to participate in the streaming architecture?

    @Anonymous User @Anonymous User @Anonymous User



    ------------------------------
    Balasubramanian Kasamuthu
    Adaptive Development
    ------------------------------