The following illustrates the updated approach introduced in v4.10, where large jobs are now processed by multiple Topology service instances using multiple Kafka partitions for improved scalability and performance.

In short, for optimal performance, run regular jobs without the Bulk-load option. Enable Bulk-load only for large jobs. Using it for every job is not recommended, as it may slow down some operations.

Since data is processed by all Topology instances, a default production-sized AIOps deployment typically includes three Cassandra nodes and two Topology service instances. If sufficient hardware resources are available, it's recommended to add a third Topology service instance, one per Cassandra node for improved performance and balanced processing.

Here’s an example: loading over 15,000 network devices using the IBM Network Manager Observer without the Bulk-load option took approximately 8 hours. With Bulk-load enabled, the observation completed in under 3 hours, resulting in a time savings of nearly 5 hours.

Last but not least, if you're using the New multi-file option for File-Observer, meaning you'll be loading multiple files—it will automatically default to bulk-load mode, unless you manually choose to disable it.