IBM webMethods Hybrid Integration allows the creation of integrations using the IBM App Connect capability. By default, the runtimes managed by App Connect can call remote TCPIP based services which are accessible on the public internet. This can include many TCPIP based protocols like HTTPS, IBM MQ, JDBC and so on. It is now possible to allow TCPIP access to private networks running in a user's own AWS account using the AWS private link mechanism. When using this the end applications do not need to be exposed to the public internet or to any other user of the webMethods Hybrid Integration system.

This allows the moving of integration logic that originally was running on user owned private networks, to run on IWHI without exposing the applications to direct attack by making them publicly available. The full benefits of using a managed system like webMethods Hybrid Integration can then be used without introducing possible attack vectors by exposing applications directly to the public network. The cost savings and efficiency of a managed system can be utilised while still having similar security to self-managed systems.

This article will take you through how to connect an instance of webMethods Hybrid Integration to a user owned AWS private network and then access this directly from an App Connect integration flow.

To illustrate how powerful and flexible the webMethods Hybrid Integration is, I will use a scenario which requires calling a service that is only available on a private network as part of its function. I will also give a full example of setting up an AWS private network using AWS private link with a running EC2 instance service.

Some of the AWS configuration used in this tutorial is done to make the tutorial easy to set up with minimal skill but for a real system you will need to consult with your own security architects to determine how you configure your own internal VPCs to comply with your own security requirements.

Scenario

A coffee shop owner tracks customer arrivals by capturing their car number plates and mapping them to user profiles. When a customer drives to the shop, their profile is retrieved, and an AI analytics tool analyses their past preferences. Based on this data, the AI tool generates personalized menu suggestions. The number plate mechanism is available in a cloud managed system, and the AI mechanism is running on an on-premises system completely self-managed by the customer.

When constructing the above scenario, the integration will need access to integrations running in both these systems.

The integration scenario will require calls to two running services.

1. The first will be a service which is running on the users AWS private network and it takes a customer number plate and converts it to a customer account. The type of service which it would be ideal to not expose directly to the internet.
2. The second service will be cloud based and provide the coffee suggestion for the customer identified by the number plate look up. For this second service, we use a callable flow running in a managed App Connect Runtime.

The system will look like:

The red route from the integration flow via the private link to the user created VPC Application is the main subject for this article. The entire route is private and not accessible by any public networking.

Prerequisites

I have included all artifacts required to set up the scenario which will be described in detail here. To follow along, and set up a similar system, you will need:

• An instance of webMethods Hybrid Integration, with the following capability enabled:
  • App Connect.
  • Private network.
• A local installation of the IBM App Connect Enterprise software product (such as the Evaluation Edition) (URL: https://www.ibm.com/resources/mrs/assets?source=swg-wmbfd)
• The following App Connect Enterprise PI file contains all the integration flows you require for this tutorial: private-link-demo-pi.zip
• An AWS account that allows the creation of:
  • VPC.
  • EC2 instances.
• Important note: you need to ensure the region your webMethods Hybrid Integration instance is in matches where you create the VPC and EC2 instances. If your instance is in eu-central-1 then create the VPCs and EC2 instance in eu-central-1.

You can follow this article without setting anything up yourself if you would like to just understand more about the webMethods Hybrid Integration product.

This tutorial will be quite long as it aims to provide details about AWS setup. If you already have an application running in AWS, you could attempt to miss out some of the 'Creating an application in EC2 to provide the number plate look up service' and try integrating with your own application instead.

Creating an application in EC2 to provide the number plate look up service.

The following setup steps will be covered in order:

• AWS setup part 1: Create a VPC and required subnets etc.
• AWS setup part 2: Create an EC2 instance to run the required service.
• AWS setup part 3: Creating the service in the EC2 instance.
• AWS setup part 4: Creating a private link endpoint service that can be used to expose the application using the AWS private link mechanism.
• AWS setup part 5: Testing the reachability of the application via the endpoint service.

Once all this is configured then setting up App Connect in IWHI to access the application can be done.

I will use the name 'ace-pl-1' in all the names of AWS components but feel free to replace this everywhere with your own naming convention.

Before starting, login to your AWS account home page which will allow you access to call the components required by the rest of the steps.

AWS setup part 1: Create a VPC and required subnets etc.

Navigate to the VPC page from AWS home.

Click on the 'Create VPC' button and select VPC and more. Fill it out as follows:

Note: We are limiting it to one AZ with 1 public and 1 private subnet for now and not creating an S3 gateway. The name is set to 'ace-pl-1'.

Click on 'Create VPC' and once creation is finished you should see:

The VPC is now created and ready for use.

AWS setup part 2: Create an EC2 instance to run the required service.

Navigate to the VPC home page and select 'Launch EC2 instances':

Call the instance 'ace-pl-1' and leave most the options to the default. For 'Key pair (login)' select an existing key pair or create a new one. You will not need to use this for the tutorial.

For the network section select the VPC 'ace-pl-1' and select the public subnet and enable Auto-assign public IP. For a real system you will probably want to use a private subnet, but this will make it easier to access the instance for this tutorial without additional network setup. We are enabling public access purely so we can log into the instance to install the application. Add an additional security rule to allow all TCP traffic from all IP addresses (for a real system you will need to be stricter on what you allow for this depending on your own security requirements):

Now click on 'Launch Instance' in the panel on the right:

Select the instance and click on the option to 'connect':

Accept the default on the connect page and you should see a command prompt.

AWS setup part 3: Creating the service in the EC2 instance

Using the command prompt from the previous step you will now need to set up the application.

For this tutorial we are going to use a simple Node.js express application.

To install the node, type the following on the command prompt:

curl -o- https://raw.githubusercontent.com/nvm-sh/nvm/v0.39.7/install.sh | bash

Now exit the command prompt and then reconnect to the instance with a new command prompt (follow the steps above from previous section on how to connect again).

nvm install --lts

Once in a new command prompt then install express:

npm install express

And finally, we need to make the express application:

vi demo-http.js

Then copy in the code found here:

demo-http.js

Save and exit vi (escape :wq)

The final step is to run the application so that it continues to run even after disconnecting from the instance:

nohup node demo-http.js &

It will start an application listening on port 3000.

You can now exit the instance again and the application will continue to run if the instance is left running.

AWS setup part 4: Creating a private link endpoint service that can be used to expose the application using the AWS private link mechanism

There are several things that need to be created to have the endpoint service as follows:

• Target groups- which identifies the instance to route traffic to.
• Load balancer - which will manage the send of traffic between the service and the instance.
• Endpoint service - which will then be connected to by IWHI to send private traffic to.

To create a target group, navigate to the EC2 home page and select 'Target group' from the left navi bar. Then select the create button:

Set the name to 'ace-pl-1' and configure to use TCP on port 3000. Also, select the correct VPC. Everything else can stay unchanged:

On the next page select and add your instance created previously:

Then just select Next and then create target group.

Once created you should see:

Now to create the load balancer. On the EC2 home page select the load balancer from the left navi bar and then select to create a load balancer. On the first screen select the 'Network Load Balancer'. On the creation screen select to be an internal load balancer and set the load balancer name to: 'ace-pl-1'. Select the correct VPC 'ace-pl-1' and the private subnet. Then also select the target group 'ace-pl-1' and the TCP port to be 3000:

Once created, you will need to edit the attributes on the load balancer to be across region. Select 'Edit load balancer attributes':

THIS STEP IS VERY IMPORTANT. IF NOT CROSS REGION IT WILL NOT WORK!

Now the load balancer is created the service endpoint can be created. Navigate to the VPC home page and select 'Create vpc endpoint service'. Call it 'ace-pl-1'. Select the ace-pl-1 load balancer:

Then select 'Create'. After a short while the service will be provisioned and ready for use in the next section.

AWS setup part 5: Testing the reachability of the application via the endpoint service.

Now we have fully set up the application and the VPC service we can use the AWS reachability tool to test if the service can route through to the instance. Navigate to the AWS VPC Reachability tool (if in doubt just search for it in the AWS search box at the top of the VPC page).

Click on 'Create and analyze path'. Select the service as the source and the instance as the destination:

Then click on 'Create and analyze path'. It will take a few minutes to complete, and you will need to refresh the Analyses until complete:

If it fails, then the most likely reasons are:

• You have not allowed the load balancer to be cross region.
• You have a security group that is blocking the traffic. The most likely security group would be the VPC security group which you need to allow TCP traffic from any source for this demo to work. 

At this point you will have a working application in an AWS VPC already for you to call via the private link from App Connect in IWHI.

Setting up IWHI to have access to the application on the private network

I will step through how to set up an outbound private network in this section using the components you have just created above. Product documentation for this can also be found for reference here: private network docs.

Navigate to the private connections page in your IWHI instance. If you do not have this then you will need to enable the capability. Once enabled you will see it in the hamburger menu in the header:

Click on create and then select to create AWS PrivateLink (Outbound).

On the first screen it asks you to have a service endpoint but this we already have from a previous step above. Click Next and then it will provide the account details which will be used to connect to your endpoint service:

Copy the ARN and then go to the VPC service endpoint we made previously. Add the ARN to the 'Allow principles'.

Go back to the IWHI UI and click the check box and Next button. It then requires your service name which you can get from VPC service definition:

Copy that to the IWHI wizard and then select Next. It will go into a state of provisioning. You will need to go back to the VPC service endpoint and accept the endpoint connection:

Now in the IWHI UI refresh until the connection becomes ready.

It is now fully configured. Select the menu at the ready entry and 'View details':

The DNS name is the host name you can now use in any integration flow to call the end application. IWHI has been configured to only allow your runtimes access to this application.

Record the hostname for use in the next section.

Creating and deploying an ACE integration which can call the private application

The integration to use for this tutorial is provided in the following ACE Toolkit PI file: private-link-demo-pi.zip

Import this into the ACE Toolkit and you should see the following projects:

One part of this needs to be modified to point at the new private link end point you have set up. Open the Test_coffee_shop project and navigate to the sub flow: 'getSuggestion' and modify the HTTP Request URL to include the DNS name recorded from the previous step:

Ensure the URL is HTTP:// and that it still has port 3000 set.

The path /number_plate_customer_lookup/v1/customer should also stay the same. In the case of this run through the DNS was: vpce-0729d0e2d6a90e162-73em5mtd.vpce-svc-0daa5219202832407.us-east-1.vpce.amazonaws.com and the URL was set to: http://vpce-0729d0e2d6a90e162-73em5mtd.vpce-svc-0daa5219202832407.us-east-1.vpce.amazonaws.com:3000/number_plate_customer_lookup/v1/customer

Once you have made the change then save the flow and go to the Test_coffee_flow BAR file and rebuild and save it. You now have the BAR file already to deploy with the correct hostname.

The next step is to create an IWHI ACE runtime and deploy the BAR. For this tutorial I will create a new runtime called 'customer-ir'. You can also reuse an existing one. In the IWHI change to the App Connect capability:

And then navigate to the manage page. Select to create a runtime and follow the wizard. Call the runtime 'customer-ir'. Select integrations and upload the test_coffee_shop.bar and select it to be deployed. Select configurations and add in 'default-switch-server-privatenetworkagent'. Select single replica and then click on Create:

You now have a running solution ready to be tested in the next section.

Running the scenario end to end

Now everything is running you can test the integration by double clicking on the customer-ir in the managed view then double click on Test_coffee_shop. In the API Explorer choose GET /Suggestions and then select 'Try it'. In the Try it view put the number plate as HF17WERT. Finally click on send:

You can also try plate HF18WERT for a different customer.

If this returns as shown in the image above then you have the scenario fully working. If it hangs then it is likely the network setup has not been done correctly somewhere in AWS. Review all the steps above to check you have it all set up correctly.

When the flow executes the following has happened:

• You make a REST call with a customer number plate to the ACE integration flow.
• The flow makes a call to the URL using private link DNS.
• The call gets routed to the VPC endpoint hosted by IWHI.
• The call gets routed to the VPC service endpoint running in your VPC.
• The service uses the load balancer to then send the call to the EC2 application which accepts the REST call and calculates the customer details.
• The flow receives the result from the call and then makes a callable flow call to another integration flow running in IWHI.
• The second flow calculates the suggestions and returns those to the main flow.
• The flow then returns the result to the original caller which you see in the API Explorer.

At this point you have fully configured and tested a system using the private link between IWHI and your own managed AWS VPC.

Details on how to extend scenario to use HTTPS rather than HTTP for calling application and other security considerations.

The tutorial does not SSL encrypt data from the ACE integration flow to the end applications. This is mainly because of the extra requirements needed to get an owned domain and a properly signed certificate. There are many ways you could look at achieving this in a real system.

One approach would be for you to have a public DNS domain that you own and are able to create properly signed certificates and keys on. You would use this to generate a new domain for your application and set it up to use the generated key. You would then need to use your public DNS system to CNAME this domain to the domain provided by the private link. You would then use your public domain in the URL from the ACE flow. This would then mean when the flow makes the REST call it would use the correct domain name for the certificate supplied by the end application but would get the private link IP address from the DNS lookup.

The tutorial does run the application on a public subnet which you probably would not do for a real isolated application, and it also opens more access than needed when configuring security groups. All of these should be considered more carefully for a real production system.

Closing thoughts

This tutorial shows how an IWHI App Connect integration flow can be securely connected to your own AWS VPC using the AWS private link technology. By doing this you can move existing integration logic from running in your own infrastructure to running in IWHI without the need to expose important systems to the public internet.

Using this you can achieve the perfect balance of security of access to your applications while starting to reduce your costs for managing integration systems.

Once you have the connectivity between IWHI and your own AWS VPC you can then leverage other AWS mechanisms to access applications in other networks including access to on-premises systems.

We only covered configuring the App Connect capability for egress from App Connect integration flow to AWS. It is also possible to do the same for ingress from AWS into a App Connect integration flow. Similar setup is also available for API Connect, webMethods API Gateway and webMethods Integration (see private network docs for full details).

Look out in the future for many more articles showing how to securely integrate with your own private systems, including calling an integration flow from a privately hosted application.