StreamSets

StreamSets

Connect with experts and peers to elevate technical expertise, solve problems and share insights.


#DataIntegration
#Data
 View Only
Expand all | Collapse all

StreamSets – Data Validation, Dynamic Source Selection, and Monitoring Capabilities

  • 1.  StreamSets – Data Validation, Dynamic Source Selection, and Monitoring Capabilities

    Posted Thu January 08, 2026 05:04 PM

    Hello IBM Team,

    We would like to confirm whether the following scenarios can be implemented using IBM StreamSets:

    1. Data volume validation
      Is it possible to validate and compare that the amount of data in both databases NRT1 and NRT2 is the same?

    2. Dynamic source selection
      Based on which database contains a larger volume of data, can the pipeline dynamically select that database as the source and replicate the data to the destination database?

    3. Multiple static IPs as source
      Is it supported to configure two static database IP addresses as sources within the same node or fragment, using the same port?

    4. Pipeline execution alerts
      Is there a way to configure automatic alerts (e.g., email notifications) when a pipeline has not been executed or fails to run as expected?

    We would appreciate your guidance or best practices related to these scenarios.

    Best regards,
    Jason



    ------------------------------
    Jason Rick Medina Arbi
    Solution Arquitect TI
    Mainsoft
    ------------------------------


  • 2.  RE: StreamSets – Data Validation, Dynamic Source Selection, and Monitoring Capabilities

    Posted Thu February 19, 2026 12:52 AM

    Hi Jason,

    1. There has to be an orchestration layer at the top to trigger the API that can run the pipeline. Connection to database can be passed as a parameter accordingly.
    2. Yes, very much possible.
    3. Yes, very much possible.
    4. Yes, definitely.

    :-)



    ------------------------------
    Saleem Pothiwala
    CEO
    Kermit Tech
    Asker, Norway
    ------------------------------