Overview

This document provides a summary of how-to setup a integration between Salesforce Change Data Capture (CDC) events, IBM webMethods Integration (SaaS), and IBM webMethods Integration Server On-Prem.

This article points to another existing article to show how to create a connected app, in Salesforce, get the client and secret, and the OAuth tokens to use in the connection.

Samples of some of the components are provided in a GitHub repo and can be imported. Other miscellaneous files are also provided, that should help if you want to put this demo together.

The instructions will guide you.

Overview of Salesforce CDC

Salesforce provides a Streaming API that enables an event-driven approach to synchronize high-volume data between systems. Instead of synchronously requesting data from Salesforce, external systems can subscribe to Salesforce events and receive data changes in real-time. This is particularly useful for integrating Salesforce with external systems.

The CDC event type is one of the most valuable event types offered by Salesforce. It captures changes to Salesforce records and sends event notifications containing the changed data. Subscribers, such as IBM webMethods Integration (SaaS), can process these events and trigger downstream actions based on the event type.

Example Use Case: When a new Account record is created in Salesforce, a CDC event is triggered. webMethods Integration (SaaS) listens for the event and processes it. Subsequently, the event data is transmitted to the on-prem webMethods Integration Server for further processing, such as storing the data in a target database.

Benefits of Using CDC Events:

·      Real-time data synchronization

·      Reduced reliance on direct Salesforce data requests

·      Improved data consistency across systems

Complexity Considerations:

·      Event delivery is not guaranteed; hence, a reconciliation process should be in place.

·      Streaming events operate asynchronously, meaning data updates may not reflect in real-time across systems. 

Note: for this demo, the CDC event will be subscribed by webMethods Integration (SaaS) and passed along to an on-premise webMethods Integration Server using Hybrid connectivity.  The on-premise webMethods service will then parse the data, and update a database. The on-prem webMethods Integration Server is not reflected in the above diagram, but should be there between External Subscriber and External Database.

Setup Salesforce CRM for Data sync

To enable data synchronization, the Salesforce CRM must be configured to generate CDC events. Follow the steps below to set up a connected app, and configure CDC events:

Make sure you have a Salesforce Developer Account
If you do not have one, signup for a free account here: https://developer.salesforce.com/signup.

1.  Complete the form and click Sign Me Up.

2.  Verify your email and activate your account. 

Create a connected app, access token and refresh token

For detailed instructions on how to create a connected app and get access/refresh tokens to use in the connection from wM Integration iPaaS tenant, refer to this IBM Community website article ----> Salesforce OAuth Configuration in webMethods.io Integration and webMethods Integration iPaaS Flow Editor   <-----IMPORTANT ! Follow this article and as you go through each step, refer also to my notes below for extra hints and clarity:

• Hint for Step 2 – Make sure you are in the Salesforce Classic UI - The above article was created using the Salesforce Classic UI. Before continuing, make sure you are in the Salesforce Classic UI. After logging into Salesforce, if you do not see the Setup menu option as shown in the screen shot (in the article), this means you are in the Lightning Experience UI. To switch to Classic UI, click the profile icon in the top right. UI, and select Switch to Salesforce Classic. 

• Hint for step 4: Scroll down on the page until you see Connected Apps section, then click the New button.

• Hint for step 5: Use your own email ID for Contact Email

• Change for step 6: I used https://jwt.ms instead of https:///www.softwareag.com

Note: The https://jwt.ms URL is used as a redirect URI for debugging and inspecting JSON Web Tokens (JWTs). It is a convenient tool provided by Microsoft to decode and view the contents of a JWT in a user-friendly interface. While it is useful for testing and development, it is not recommended for production use. In a production environment, the redirect URI should point to a valid endpoint in your application where the authorization code or token can be securely handled.    

• Hint for step 7: In the section API (Enable OAuth Settings), I also unchecked (disabled) the following setting for “Require Proof Key for Code Exchange (PKCE)....”. I had to do this since my authorization flow was not working. Rather than play around with PKCE configuration in Salesforce for a demo example, I chose to disable it. See Note below.

Note: PKCE stands for Proof Key for Code Exchange. It should be used in OAuth authentication to enhance security, especially in public clients like mobile or single-page applications where the client secret cannot be securely stored. It mitigates the risk of authorization code interception during the OAuth flow by requiring a code challenge and verifier. While PKCE is optional in some scenarios, it is recommended for all OAuth flows to ensure stronger security, particularly in production environments. For demo or development purposes, it can be disabled if simplicity is prioritized.

• Hint for step 9: If you are not able to find the Consumer key and consumer secret for the connected app, make sure you are logged into your Salesforce tenant, then click Setup. In the search box, type build, then select Create 

Scroll down again until you find Create --> Apps (in the Build section in the left side nav bar) You will then see a list of Apps.

Scroll down until you find the app you created in the Connected Apps section. Click on the link (below Connected App Name) to open the app.

In the API (Enable OAuth Settings) section, click Manage Consumer Details (you may be asked to verify your identity by entering a verification code). You can then view the Consumer Key and Consumer Secret. Copy both these values somewhere (store these somewhere for later reference. You will need them in upcoming steps).

• Hint for step 10: Copy your entire domain name. For example:

Instead of using https://ap15.salesforce.com I used https://orgfarm-fb41715d3b-dev-ed.develop.my.salesforce.com/ along with the other required parameters.

Use the Consumer key value you saved and paste it into {consumer_key}.

Replace {callback_url} with the callback you set when configuring the connected app (e.g. https://jwt.ms)

Note: This is my example. Your domain name will be different.

•    Hint for step 13: your characters at the end may be different. If you see % characters at before other characters, then it is encoded and needs to be decoded. For example, my authorization code ended in: %3D%3D

So I had to run the utility mentioned to decode.

·         Hint for step 14: I used https://orgfarm-fb41715d3b-dev-ed.develop.my.salesforce.com/ instead of https://<instance_name>.salesforce.com/ As mentioned, your URL will be different. * For the {authorization_code} use the decoded value.

·         Step 16: I did this step but did not do step 17 and beyond. For the rest of this Salesforce Data Sync demo you can continue with the next section below: Enable CDC Events in Salesforce.         

Enable CDC Events in Salesforce

1.  In Salesforce, navigate to Setup.

2.  Search for Change Data Capture.

3.  Select the Salesforce objects you want to track. In this demo I only selected Account.

4.  Save your changes.

Setup IBM webMethods integration (iPaaS) to listen for SF CDC Events

Create Flow Service to handle CDC Events and Errors

The un-happy path Flow Service will be called automatically if there are issues with the Listener. The example service will:

·         Receive the error data from the listener in the input pipeline, converts it to JSON strings, and then writes them to a log.

·         For an example that also sends an email through SMTP, refer to this doc: https://www.ibm.com/docs/en/wm-integration?topic=listeners-how-configure-error-callback-service-spec-flow-service-salesforce-listener

IMPORT EXAMPLES ---> You can import example (demo) Flow Services into your webMethods Integration project by importing the zip files HandleSalesforceCDCevent.zip and SFchangeListenerErrorCallbackFlowSvc.zip
from my GitHub repo here: https://github.com/WayneLeishman/ExampleFlowServicesSFdataSync

·         After importing, review both services.

Setup your listener (trigger) in wM Integration iPaaS

In the Listener section, click the New listener button. Currently only Salesforce listeners are supported. Select Salesforce as the type of listener and select the Salesforce CRM REST version.

·         You will then continue to configure the listener:

•         For the Streaming API endpoint URL, use your own Salesforce tenant but keep a suffix of /cometd/60.0 (or whatever version you are using).

•         For the replay option I just selected ALL. In a real solution you will need to determine the setting based on what is required (consult the documentation).

•         In the Select Account section, if you do not have a Salesforce account to use, click the plus sign.

§  Set Authorization Type as OAuth V2.0 (Authorization Code Flow) and OAuth Type as Custom.

§  Provide the info to connect to your app in Salesforce. Fill in the OAuth details as shown below:
Note: Set Grant Type to refresh_token.
Instance and Server URL are mine. Use your own Salesforce tenant name.
I just left the other values at their default.
Click Save when you are done.

•         After adding the Account connection, set subscription to “Salesforce Change Data Capture Event. This will ensure the listener receives the identified events when changes happen in Salesforce. A real solution may require other subscription types. Multiple listeners are allowed.

•         Set Parameters as shown below (remember, these are examples used in the demo. In a real solution check the documentation for info on all fields).

Note: In this listener we are listening for ChangeEvents. This will ensure the listener receives the identified events when changes happen in Salesforce. A real solution may require other subscription types (Multiple listeners are allowed).

•         Configure the Listener Event settings so that the listener knows which services to invoke.
Select the implemented Flow Service names for success, and a service to handle callback errors:
(in the screen shot below I did not pre-create the Callback Flow service but I was able to create a basic callback handler by clicking the plus sign. If you imported the sample Flow Services into webMethods Integration iPaaS then the callback service will already be there.

•         Select a user to run the Flow Services.

•         Here’s my configured Event values:

•         Finally, review the listener configuration on the Summary panel. Click Done when you are finished.

•         Make sure your Listener is enabled.

Handle SF CDC Events in wM On-Premise

The final step is to process the CDC event data in the on-premise webMethods Integration Server (IS). The goal is to sync the Salesforce event data into a target system (e.g., database, file system).

Step 1: Receive the Event Data in your on-premises environment

1.      Expose webMethods IS services within your IS package(s) to accept incoming CDC event data. Here’s the services I exposed:
   
   
   Here’s my application (package):
   

2.  Define an input signature that matches the CDC event payload.

NOTE: I created the Input document type for CDC by a) reading the Salesforce documentation b) trapping a CDC event and exploring the format.
You can download the JSON test files I used to create the IS docs and to do my unit testing from my GitHub repo here: https://github.com/WayneLeishman/ExampleFlowServicesSFdataSync/blob/main/Salesforce%20CDC%20Events%20test%20data.zip
(I only implemented Create and Delete for the demo in my IS Package).

Step 2: Process the Event Data

Use an IS Service to handle different event types based on the changeType field in the event payload. If the event changeType was equal to CREATE, then I inserted it in a database table. If changeType was DELETE, then delete it from the DB table.
Example actions:

·      CREATE: Insert a new row in the database.

·      UPDATE: Update the existing row.  (I did not implement this for the demo).

·      DELETE: Delete the corresponding row.

Step 3: Integrate with Target System

•         Since I put my data in a DB table, I used the webMethods adapter for JDBC to integrate with the target database.
  Note: I used an on-prem PostgreSQL database but you can use whatever you want. In the GitHub repo, I provided the init.sql that I used to create two tables.

Note: You can get the webMethods on-premise IS package from the GitHub repo mentioned above. The package is in ExpertImpl_SFsyncToOnPrem.zip. You can install this as a regular IS package. Note: my example uses the wM JDBC adapter.

Summary

This article outlined a complete use case for event-driven data synchronization between Salesforce and an on-premise webMethods Integration Server. By leveraging Salesforce CDC events and webMethods Integration (SaaS), organizations can achieve real-time data updates and minimize dependency on synchronous API requests.

The hybrid integration model demonstrated here showcases how powerful event-driven architectures can be when integrating cloud and on-premise systems.