Hi Team,

We have reviewed the StreamSets
documentation for Pipeline Finisher
Executor and noticed that Kafka
Consumer is NOT listed as a
supported origin for no-more-data
events.

We are trying to:
1. Read all existing records
   from Kafka topic
2. Load to Snowflake
3. Stop pipeline automatically
   when all records are loaded

We have tried:
1. Pipeline Finisher with
   precondition:
   ${record:eventType() ==
   'no-more-data'}
   → Not working!

2. Connecting Pipeline Finisher
   via event output
   → Lightning bolt not visible
      in Kafka Consumer stage

3. Max Empty Batches setting
   → Not available in our version

Could you please advise:
1. Is there a supported way to
   auto stop Kafka Consumer
   pipeline after reading
   all existing records?

2. What is recommended approach
   for one time Kafka to
   Snowflake load?

3. Should we use a different
   origin or approach?

Thank you!

Hi Rahul,

Options include using the Scheduler, an Orchestration Pipeline, the REST API, Metric Rules, Alerts and webHooks.   

Example:  A "Pipeline is Idle" Metric Rule, modified to 12 seconds, and the Job Stop REST call as a webHook: /jobrunner/rest/v1/job/${JOB_ID}/stop.

Regards,

------------------------------
Eric Greisdorf
------------------------------