Hello my deloved IBM streamsets users,

I have been stuck with another issue on fetching data from sqlserver to snowflake. These are full loads and i am using JDBC consumer to have a batch size. I have kept the max batch size as 5000, the total rows that i am trying fetch and load is 100k. The issue is, the pipeline runs only for the first 5000 rows and it finishes. Not sure what i am missing, i have changed the JDBC connector to have responseBuffering=adaptive attached to as per recommendation from microsoft. But it didnt fix the issue. My expectation is to have multiple batches run when i trigger this pipeline. what am i missing? your help are always appreciated.

Thanks

Vivek TP

Hi Vivek,

Would you like to have a working session? Please contact me at eric.greisdorf@ibm.com, or book time time directly on my calendar here [link]. 

This is a very popular use case (Query and MultiTable). The issue is most likely with a pipeline configuration, and not on the SQL Server side.

Please check that the Origin has events enabled, and the Pipeline Finisher configuration.

It should be set to finish on the Origin's no-more-data event.

Screenshots below.

Best Regards,

Eric