StreamSets and webMethods

About the Article

This article explains different ways to connect StreamSets with webMethods On Prem products like Universal Messaging and Integration Server.

1.0 StreamSets Landscape

Any development platform will have 3 main components: Design time components, Runtime Components and Administration/Control components. Sometimes Design time components and Administration Components are merged. Streamssets follows the same path. Its Admin component that is “Control hub” is also its Design time component.

Runtime component can be hosted on Cloud providers (AWS, Azure, GCP) or locally. Once a Deployment set is defined on the Control hub, it allows users to download the deployment script. Either as a tar ball or a docker command. Running instance of a Deployment is called and engine.

To Engine automatically connects to the control hub and users can manage the engine from the control hub. Since StreamSets engines can be executed anywhere, to work with webMethods the StreamSets engine and webMethods instances should be reachable to each other.

A StreamSets Data pipeline (aka Workflow ) consists of mainly 3 types of steps.
• A Data Origin to read data from
• A Data Destination to send data to at the end
• Processing steps are Intermediary between Source and Destination which processes data.

2.0 Messaging

For messaging StreamSets provides connectors for different messaging technologies like JMS, Kafka, Amazon SQS, Azure Event Hub, Apache Pulsar etc.

To connect with Universal Messaging we can use JMS Consumer as Origin and JMS producer as Destination.

To use JMS tools should be enabled and target JMS library should be added in the Engine.

2.1 Enable JMS Stage Library in the Deployment

Goto Setup > Deployments

Edit the required Deployment

Goto Stage 2

Click on Stage Libraries

Search for JMS, this will list JMS Consumer and JMS Producer
Click add all to move them to Selected stages pane on the right

Click ok and click Save & Next
Goto Last section and Restart engine.

2.2 Add Universal Messaging JMS Client libraries to the Engine

Goto Setup > Engines

Click on the required Engine

In Edit page, Goto External Resources > External Libraries

Click on ‘+’ icon and browse UM JMS client jars nJMS.jar and nClient.jar from local SoftwareAG installations.

<SoftwareAG>/UniversalMessaging/lib/nClient.jar

<SoftwareAG>/UniversalMessaging/lib/nJMS.jar

After upload Click Restart Engine to restart engine

2.3 Create a Connection

Goto Setup > Connections

Click on + icon to add a new Connection

Give it a Name and Description

Select Authoring Engine as the engine where JMS libraries are deployed

For Type, Select JMS

Click Save & Next to go to Configure Connection section

Add Universal Messaging connection details

JMS Initial Context Factory: com.pcbsys.nirvana.nSpace.NirvanaContextFactory

JNDI Connection Factory: Connection factory name from UM

JMS Provider URL: UM URL nsp://<host>:<port>

Click on Test Connection, this should show Green Tick at the top

Click Save

2.4 Use JMS stages in Data Pipelines

Goto Build > Pipelines

Create a Pipeline

Add a origin to read data from.

In the Destination drop down, select JMS Producer.

Leave defaults in the General Tab

Goto JMS tab

For Connection select the connection created in previous section

Configure JMS Destination name and JMS Destination type as per configuration from UM

In Data Format tab,
Select a suitable data format. Example JSON.

2.5 Results

Start the pipeline and see results,

3.0 REST API

StreamSets provides HTTP Client Destination, which can be used to make REST API Calls.
This feature can be used to call REST APIs on SofwareAG products Integration Server, API Gateway, webmethods.io Integration, webmethods.io B2B etc.

3.1 Use HTTP client in Data pipelines

To call an API on Integration Server, use HTTP Client in Data pipeline

3.2 Configure HTTP Endpoint details

Set up REST Endpoint details

Set Credentials in Credentials tab and set appropriate Data format in Data Format tab

3.2 Results

Run the pipeline to see results on StreamSets control hub and Integration Server logs

Useful links | Relevant resources

https://academy.streamsets.com/courses/dataops-platform-fundamentals/

https://academy.streamsets.com/courses/certification-onboarding/