StreamSets

StreamSets

Connect with experts and peers to elevate technical expertise, solve problems and share insights.


#DataIntegration
#Data
 View Only
Back to discussions
Expand all | Collapse all

StreamSets DataOps Platform - Auto Stop Pipeline Configuration for Kafka to Snowflake One Time Load

  • 1.  StreamSets DataOps Platform - Auto Stop Pipeline Configuration for Kafka to Snowflake One Time Load

    Posted 6 days ago
    Edited by Rahul Dharmawat 6 days ago     
    Hi Team,

We have reviewed the StreamSets
documentation for Pipeline Finisher
Executor and noticed that Kafka
Consumer is NOT listed as a
supported origin for no-more-data
events.

We are trying to:
1. Read all existing records
   from Kafka topic
2. Load to Snowflake
3. Stop pipeline automatically
   when all records are loaded

We have tried:
1. Pipeline Finisher with
   precondition:
   ${record:eventType() ==
   'no-more-data'}
   → Not working!

2. Connecting Pipeline Finisher
   via event output
   → Lightning bolt not visible
      in Kafka Consumer stage

3. Max Empty Batches setting
   → Not available in our version

Could you please advise:
1. Is there a supported way to
   auto stop Kafka Consumer
   pipeline after reading
   all existing records?

2. What is recommended approach
   for one time Kafka to
   Snowflake load?

3. Should we use a different
   origin or approach?

Thank you!


  • 2.  RE: StreamSets DataOps Platform - Auto Stop Pipeline Configuration for Kafka to Snowflake One Time Load

    Posted 5 days ago
    Edited by Eric Greisdorf 2 days ago

    Hi Rahul,

    Options include using the Scheduler, an Orchestration Pipeline, the REST API, Metric Rules, Alerts and webHooks.   

    Example:  A "Pipeline is Idle" Metric Rule, modified to 12 seconds, and the Job Stop REST call as a webHook: /jobrunner/rest/v1/job/${JOB_ID}/stop.

    Regards,



    ------------------------------
    Eric Greisdorf
    ------------------------------

    Original Message


Related Content