App Connect Enterprise’s flow debugger is widely used in flow development as it enables rapid problem resolution using local servers: the solution can be deployed, tested, and debugged from the toolkit. Moving flows to ACE-as-a-Service (ACEaaS) in AWS appears to break this approach, in that the servers are not visible on the same network as the toolkit and so the debugger will not be able to connect directly, but debugging is still possible using the “private network connection” capability of the ACE service.

Using a debugger on a cloud server does require some care, especially if there are multiple replicas running at once, but can be useful in the early stages of moving flows to the cloud because it may be the simplest way to solve cloud-specific configuration issues. Other techniques such as flow logging or monitoring should be used for ongoing problem detection and management (not covered here).

Quick summary: run a switch client on the toolkit system and reverse the usual switch connections to use an endpoint in the cloud.

Background

The toolkit flow debugger uses the Java Debug Wire Protocol (JDWP) to connect to an ACE server, relying on the standard Java debugging support with some ACE-specific extensions to allow breakpoints in flows, message tree visibility, ESQL line-by-line debugging, etc. The picture looks as follows when running locally or using a remote server on the same network:

Because JDWP is using TCPIP as a transport, any port-forwarding technology can be used to enable connectivity, including SSH, kubectl, and the ACE switch client/server; the only requirement is that the TCPIP connection succeed, and that the remote server is actually running the flows in the toolkit workspace (mismatched source can cause issues). 

For ACEaaS, the ACE switch server is already provided and the other options would be difficult to configure, so using the switch client and server is the best way forward. There is currently no automatic setup capability for remote debugging as there was in the past (see Remotely debug your enterprise integrations in IBM App Connect ) but it is still possible to connect with a bit of setup. Although debugger connections can be achieved this way, server connections on port 7600 are not possible, so the ACEaaS console must be used instead to deploy flows and perform other administrative tasks.

Note that Kubernetes and CP4i users can use a similar approach as described at Remote Debugging With App Connect on Kubernetes without needing to use the switch.

Solution Overview

The switch client/server model relies on having a switch server running in a network-visible location (that is, accessible via TCPIP from all environments to be connected), with switch clients connecting into this one central server. Once connected, the switch clients can establish secure connection tunnels (TCPIP port forwarding) and also invoke callable flows on other servers. In this case, connection forwarding is the only aspect to be used, as that enables the debugging connections from the ACE toolkit to the remote server.

The toolkit is told to connect to localhost with a specific port number, and this connection is then forwarded via the switch so it can connect to the debug port on the server in the cloud. This is all at the application level, without needing any IP packet rewriting (such as iptables) to achieve the results.

The two ends of the switch links have matching configurations, with the cloud side specifying “endpoints” and the laptop end “listeners”:

When the laptop switch client comes in with a request to connect to “127.0.0.1:12345”, the switch server knows where to send the request because the cloud server has registered a matching endpoint.

This is effectively reversing the standard approach documented at https://www.ibm.com/docs/en/app-connect/saas?topic=designer-connecting-private-network but the switch server will match listeners to endpoints regardless of direction.

Configuration

To enable flow debugging, a server.conf.yaml configuration is needed with the following contents:

ResourceManagers:
  JVM:
    jvmDebugPort: 12345

The port number is arbitrary (though it should be above 1024 and not conflict with other services) but the same port should be used in all configurations if possible to avoid confusion. This port number should be unique for a particular integration runtime, too, as having multiple runtimes registering the same port (in the next section) may trigger load-balancing behavior and make it hard to predict which server is being debugged.

See the “Private network connections” tab on the App Connect SaaS console for details on creating network configurations; the default switch server network configuration cannot be changed, and so a new private network configuration will be needed for the on-cloud setup to be possible:

The “View setup instructions” option under “Actions” can be used to retrieve the JSON configuration for the new configuration, or the JSON can be copied from the "Configurations" page using the "Update" option:

Regardless of the approach taken, the JSON will need to be updated as follows, with the cloud-side JSON then placed into the ACEaaS configuration using "Update".

The laptop switchclient.json needs to specify the destination host as 127.0.0.1 (not localhost, which does not seem to work in all cases) and the port from the server.conf.yaml above for remote and local:

  "listeners": [
    {
      "localPort": 12345,
      "remoteHostname": "127.0.0.1",
      "remotePort": 12345
    }
  ]

The on-cloud server configuration (of “Private Network Agent” configuration type) must have a switchclient.json with matching remote host and port:

  "endpoints": [
    {
      "name": "debugPort",
      "hostname": "127.0.0.1",
      "port": 12345,
      "useTLS": false,
      "certs": {
        "rejectUnauthorized": true
      }
    }
  ]

Note that it is possible to assign a separate local port number to the laptop switch client and point the toolkit at that port, but this may be more confusing than just leaving the port the same everywhere.  

Example configuration files

switchclient.json for laptop example:

{
  "id": " ac498d6f-9306-4bc4-8308-32187c5ff586",
  "admin": "enabled",
  "callableFlows": "enabled",
  "listeners": [
    {
      "localPort": 12345,
      "remoteHostname": "127.0.0.1",
      "remotePort": 12345
    }
  ],
  "switch": {
    "url": "wss://default-switch-server-switch-acuyeay0evh.switch.p-vir-d1.appconnect.ibmappdomain.cloud:443",
    "certs": {
      "ca": [
        "-----BEGIN CERTIFICATE-----\nREDACTED\n-----END CERTIFICATE-----\n"
      ],
      "cert": "-----BEGIN CERTIFICATE-----\nREDACTED\n-----END CERTIFICATE-----\n",
      "key": "-----BEGIN PRIVATE KEY-----\nREDACTED\n-----END PRIVATE KEY-----\n",
      "rejectUnauthorized": true
    }
  },
  "displayName": "pnc-tdolby-thu-22-feb-2024-03-36-02-gmt-720"
}

Cloud private network configuration example:

{
  "id": " ac498d6f-9306-4bc4-8308-32187c5ff586",
  "admin": "enabled",
  "callableFlows": "enabled",
  "endpoints": [
    {
      "name": "debugPort",
      "hostname": "127.0.0.1",
      "port": 12345,
      "useTLS": false,
      "certs": {
        "rejectUnauthorized": true
      }
    }
  ],
  "switch": {
    "url": "wss://default-switch-server-switch-acuyeay0evh.switch.p-vir-d1.appconnect.ibmappdomain.cloud:443",
    "certs": {
      "ca": [
        "-----BEGIN CERTIFICATE-----\nREDACTED\n-----END CERTIFICATE-----\n"
      ],
      "cert": "-----BEGIN CERTIFICATE-----\nREDACTED\n-----END CERTIFICATE-----\n",
      "key": "-----BEGIN PRIVATE KEY-----\nREDACTED\n-----END PRIVATE KEY-----\n",
      "rejectUnauthorized": true
    }
  },
  "displayName": "pnc-tdolby-thu-22-feb-2024-03-36-02-gmt-720"
}

Warnings in startup logs

The ACE certified container startup may show a warning about “endpoints”, but this can be ignored:

WARNING: A privatenetworkagent configuration which includes 'endpoints' has been attached. Ensure that the 'endpoints' are available to be called directly from this runtime. If the 'endpoints' are not available to be called from this runtime then remove them from the configuration.

Laptop configuration

Once the switch client in the cloud is running and the debug port has been set, the laptop switch client can be configured with the switch client JSON.

This can be achieved using the secure agent downloaded from the cloud service or via an independent integration server. For the latter, all that is needed is a work directory for the server, and the switchclient.json (see above) to be placed in the <work-dir>/config/switch directory. The server should connect on startup and display a message:

2024-02-21 21:43:08.443881: BIP6464I: The secure connectivity agent for cloud systems has established a connection to the Switch server with URL 'wss://default-switch-server-switch-ac2vkpa0udw.apps.appc-p-vir-d1.pngv.p1.openshiftapps.com:443'.

The secure client provides more detailed logging and can be pointed at the configuration file without it needing to be in a specific directory:

secureagent.exe -c c:\tmp\switchclient.json -l c:\tmp\logs --accept-license-silently

Toolkit

Once the switch client is running locally and has connected successfully, the toolkit can be pointed at the client port. This requires a new "Debug Configuration" that can be created from the "Debug" perspective:

with the correct host and port entered into the new configuration:

In this case, localhost will normally work, but 127.0.0.1 could be used also.

Clicking on the "Debug" button should then cause the toolkit to connect, and at that point debugging will work as normal.

Summary

Debuggers can make solving certain problems much quicker, and this is especially true when working with cloud credentials and other configuration. While it might be helpful if ACEaaS provided an out-of-the-box solution that allowed a debugger to be attached, the approach described above should be sufficient for most current use cases.

Acknowledgements

Thanks to Muhammad Islam for his assistance in the preparation of this post.