#Kafka #migration #IBMBob #performanceoptimization #AI

Introduction 

About the migration

This migration was completed by our engineering team with assistance from Bob, our AI-powered development assistant. Throughout this post, references to “Bob” refer to the AI tool that supported the migration effort by accelerating tasks such as code analysis, dependency resolution, and implementation guidance. In the “How Bob helped” section, we’ll cover both the advantages and the limitations of incorporating AI assistance into the development workflow.

Why migrate?

Our Kafka Connect handles millions of events daily, streaming data from Kafka topics into OpenSearch for real-time analytics. Kafka Connect simplifies this workflow with built-in offset management, automatic workload distribution, and configuration-driven deployment—avoiding the need for custom consumer applications.

However, continuing to rely on the deprecated High Level REST Client (HLRC) posed growing risks and limitations. Comparing HLRC with the OpenSearch Java Client highlights why migration was essential:

By comparing the two, it’s clear that HLRC was no longer suitable for high-throughput, scalable pipelines. The OpenSearch Java Client not only resolves technical debt but also improves performance, reliability, and maintainability—ensuring our Kafka Connect pipeline is future-ready.

Architecture Overview

Step-by-step gradle migration

1. Update dependencies

Remove:

implementation 'org.opensearch.client:opensearch-rest-high-level-client:2.19.1' // HLRC 2.19.1

Add:

dependencies {
    // OpenSearch Java Client
    implementation 'org.opensearch.client:opensearch-java:3.2.0'
    
    // Apache HttpClient 5 (note: client5, not client)
    implementation 'org.apache.httpcomponents.client5:httpclient5:5.3.1'
    implementation 'org.apache.httpcomponents.core5:httpcore5:5.2.4'
    
    // JSON processing (required)
    implementation 'com.fasterxml.jackson.core:jackson-databind:2.17.0'
}

2. Handle conflicts

configurations.all {
    exclude group: 'org.apache.httpcomponents', module: 'httpclient'
    exclude group: 'org.apache.httpcomponents', module: 'httpcore'
}

3. Verify

./gradlew dependencies --configuration runtimeClasspath
./gradlew clean build

Maven alternative:

<dependency>
    <groupId>org.opensearch.client</groupId>
    <artifactId>opensearch-java</artifactId>
    <version>3.2.0</version>
</dependency>

Key code changes

1. Client initialization

Before (HLRC):

RestHighLevelClient client = new RestHighLevelClient(
    RestClient.builder(new HttpHost("localhost", 9200, "https"))
);

After (java client):

ApacheHttpClient5TransportBuilder builder = 
    ApacheHttpClient5TransportBuilder.builder(
        new HttpHost("https", "localhost", 9200)
    );

OpenSearchTransport transport = builder.build();
OpenSearchClient client = new OpenSearchClient(transport);

2. TLS configuration

Before:

// HttpClient 4
import org.apache.http.impl.nio.client.HttpAsyncClientBuilder;

After:

// HttpClient 5 - note package change
import org.apache.hc.client5.http.impl.nio.PoolingAsyncClientConnectionManager;
import org.apache.hc.client5.http.ssl.ClientTlsStrategyBuilder;

SSLContext sslContext = SSLContexts.custom()
    .loadTrustMaterial(trustStore, null)
    .build();

ClientTlsStrategy tlsStrategy = ClientTlsStrategyBuilder.create()
    .setSslContext(sslContext)
    .build();

3. Bulk operations

Before (HLRC):

BulkRequest bulkRequest = new BulkRequest();
IndexRequest indexRequest = new IndexRequest(index)
    .id(id)
    .source(jsonString, XContentType.JSON);
bulkRequest.add(indexRequest);
BulkResponse response = client.bulk(bulkRequest, RequestOptions.DEFAULT);

After (java client):

// Parse JSON to Map (type-safe)
Map<String, Object> docMap = OBJECT_MAPPER.readValue(
    payload, 
    new TypeReference<Map<String, Object>>() {}
);

// Build typed operation
BulkOperation operation = new BulkOperation.Builder()
    .index(i -> i
        .index(index)
        .id(id)
        .document(docMap)
    )
    .build();

// Execute
BulkResponse response = client.bulk(b -> b.operations(List.of(operation)));

Challenges

Challenge 1: JSON parsing

Problem: HLRC allowed indexing raw JSON strings directly, while the OpenSearch Java Client expects structured Java objects such as Map<String, Object>.

Solution: We used Jackson’s ObjectMapper to convert JSON payloads into typed Java structures before sending them to OpenSearch.

private static Map<String, Object> toDocMap(String payload) {
    return OBJECT_MAPPER.readValue(payload, 
        new TypeReference<Map<String, Object>>() {});
}

Impact: This improved type safety and validation during request construction, helping catch malformed payloads earlier in the development lifecycle instead of failing at runtime.

Challenge 2: Error handling

Problem: The error handling model changed completely in the OpenSearch Java Client. Existing logic based on hasFailures() was no longer compatible.

Solution: We migrated to the new structured error handling approach using errors() and item-level error inspection.

if (bulkResponse.errors()) {
    bulkResponse.items().forEach(item -> {
        if (item.error() != null) {
            LOGGER.error("Failed: {}", item.error().reason());
        }
    });
}

Impact: The new client exposed richer error metadata such as error type, reason, and status code, making debugging and operational troubleshooting significantly easier.

Challenge 3: Authentication

Problem: Apache HttpClient 5 introduced updated authentication APIs and changed credential handling patterns.

Solution: We updated the connector to use the new BasicCredentialsProvider implementation and adapted password handling to use char[].

BasicCredentialsProvider credentialsProvider = new BasicCredentialsProvider();
credentialsProvider.setCredentials(
    new AuthScope(host, port),
    new UsernamePasswordCredentials(username, password.toCharArray())
);

builder.setDefaultCredentialsProvider(credentialsProvider);

Impact: The migration aligned the connector with the newer HttpClient APIs while also improving credential security by avoiding immutable String objects for password storage.

Challenge 4: Bulk processing regression

Problem: During the migration, bulk writes were unintentionally disabled, causing events to be sent one-by-one instead of in batches of 200. This became the most critical performance issue we encountered and reinforced the importance of thorough performance testing and observability.

Impact:

Root cause:

The migration correctly built BulkOperation objects, but during implementation with Bob’s assistance, a logic oversight led to requests being executed immediately inside the loop instead of being accumulated into batches.

Incorrect implementation:

for (SinkRecord record : records) {
    BulkOperation op = buildOperation(record);
    // Sending immediately - NO BATCHING!
    client.bulk(b -> b.operations(List.of(op)));
}

Correct:

List<BulkOperation> batch = new ArrayList<>();

for (SinkRecord record : records) {
    batch.add(buildOperation(record));
    
    // Send when batch is full
    if (batch.size() >= 200) {
        client.bulk(b -> b.operations(batch));
        batch.clear();
    }
}

// Send remaining
if (!batch.isEmpty()) {
    client.bulk(b -> b.operations(batch));
}

Solution - BulkProcessor:

public class BulkProcessor {
    private final OpenSearchClient client;
    private final Deque<BulkOperation> queue;
    private final int batchSize = 200;
    
    public void enqueue(BulkOperation op) {
        queue.add(op);
        if (queue.size() >= batchSize) {
            flush();
        }
    }
    
    public void flush() {
        if (queue.isEmpty()) return;
        
        List<BulkOperation> ops = new ArrayList<>(queue);
        client.bulk(b -> b.operations(ops));
        queue.clear();
    }
}

Key lessons:

• Performance testing is CRITICAL

• Test with realistic data volumes

• Monitor batch sizes in production

• Benchmark before and after

• Don't assume batching is automatic

Performance results

Migration checklist

• Update Gradle/Maven dependencies

• Replace RestClient

• Update TLS configuration

• Migrate authentication

• Convert JSON strings to Map

• Implement proper bulk batching

• Update error handling

• Add data stream support

• Update integration tests

• Performance test with production-like volumes

• Monitor batch sizes

How Bob helped

Our AI assistant Bob played a significant role in accelerating the migration from HLRC to the OpenSearch Java Client. It acted as both a guide and a productivity booster across multiple stages of the migration.

1. Codebase analysis and planning

Bob helped kickstart the migration by scanning the codebase and identifying all HLRC usage patterns. It highlighted impacted files, categorized them by complexity, and provided a rough effort estimate—giving us a clear and structured starting point.

Impact: Reduced manual analysis effort and ensured no critical components were missed.

2. Dependency and compatibility resolution

Handling dependency conflicts—especially between HttpClient 4 and 5—was one of the trickiest parts. Bob provided precise dependency updates and conflict resolution strategies, eliminating trial-and-error.

Impact: Saved hours of debugging and avoided runtime classpath issues.

3. API migration guidance

From client initialization to bulk operations, Bob guided the transition to the new type-safe APIs. It also highlighted important differences such as:

• JSON handling (String → Map<String, Object>)
• Updated request/response models
• Changes in error handling patterns

Impact: Accelerated code migration and reduced learning curve for the new client.

4. TLS and authentication setup

The shift to HttpClient 5 introduced major changes in SSL and authentication configuration. Bob provided working examples for TLS setup and credential handling, including subtle but critical details (like using char[] for passwords).

Impact: Quickly resolved issues that typically require deep debugging.

5. Testing and validation strategy

Bob emphasized the importance of testing beyond unit tests and suggested:

• Integration tests with real OpenSearch instances
• Performance benchmarking with large datasets
• Validation of batch sizes and throughput

Impact: Helped establish a more robust testing approach—though gaps still remained.

6. Performance benchmarking

Bob provided templates and guidance for measuring throughput, latency, and memory usage, enabling us to validate improvements quantitatively.

Impact: Enabled data-driven comparison between HLRC and the new client.

Where Bob fell short

Despite its advantages, Bob wasn’t a silver bullet:

• A critical batching issue still made it to production
• Some suggestions were accepted without sufficient validation
• It lacked visibility into real production behavior and constraints

Key takeaways

• Always benchmark with production-like data volumes — Synthetic tests rarely expose the performance bottlenecks and edge cases that appear in real-world environments.
• Implement comprehensive monitoring before migration — Observability is essential for detecting regressions, performance issues, and unexpected behavior early.
• Use feature flags for gradual rollouts — Incremental deployment strategies make it easier to validate changes safely and roll back quickly if needed.
• Document every configuration change — Detailed records are invaluable for debugging, auditing, and maintaining operational consistency.
• AI tools accelerate development but do not replace testing — Bob significantly improved development speed and assisted with repetitive tasks, but thorough human review and validation remained critical throughout the migration process.

Conclusion

The migration to the OpenSearch Java Client delivered the expected gains—better performance, improved resource efficiency, and a more maintainable, type-safe API. But more importantly, it reinforced a deeper lesson: migrations change system behavior, not just code.

From dependency conflicts to API differences and performance pitfalls, every layer required validation under real workloads. The batching issue was just one example of how small implementation details can have outsized impact in high-throughput systems.

AI tools like Bob helped us move faster, but they didn’t replace the need for testing, observability, and engineering judgment.

In the end, the real success wasn’t just the migration—it was building a stronger approach to performance validation and system reliability at scale.

Resources

• OpenSearch Java Client Documentation

• Apache HttpClient 5 Migration Guide

• Kafka Connect Documentation

• OpenSearch Bulk API Best Practices