Hi Team,
We have a live Kafka system with fluctuating record volumes and we want to load data into Snowflake using StreamSets microbatches. What is your recommended approach for reliability, performance and error recovery?

Hi Rahul,

Our StreamSets' Snowflake Destination documentation has several out of the box options, including micro-batches. 

I also suggest the Loosely Coupled Architecture Course  and this video introduction about StreamSets Kafka Pipelines.

The Snowflake ingest process generally depends on the message volume, format, velocity, SLAs and in-flight transformations.

With fluctuating message volumes, a loosely coupled multi-pipeline approach is often suggested (it depends).

Best Regards,

Eric