products versions - {webMethods Integration 11.0 or above}
Introduction
Develop Anywhere, Deploy Anywhere introduces a new unified integration platform for webMethods and builds on the strengths of our existing products.
In this article, I will explain how you can build a readymade image for running your edge runtime container. This avoids the requirement for the edge runtime to synchronize each time it starts up and is our recommended practice for production environments.
Pre-requisites
- webMethods.io tenant that has these new features enabled.
- wpm cli tools to allow you to generate the required Dockerfiles(s).
- git cli tools to allow you fetch the example webMethod integration project.
- GitHub account to allow you to pull packages without rate limiting.
- IBMid that allows you to download Software AG licensed packages.
Steps to follow
As part of the series of articles on Develop anywhere, Deploy anywhere I previously wrote about how you can build an integration based on webMethods Packages (Develop anywhere - A practical guide to using packages with webMethods.io). Now we will look at how you can deploy this integration into your production environment, without the container having to sync the packages at startup.
Syncing is a powerful feature to speed development and testing but is not necessarily a good fit in a production environment where you want to take advantage of rapid auto-scaling. Generally, it is considered good practice for containers to be based on pre-baked images, thus ensuring consistent behavior when scaling containers or restarting them.
In the following sections, we will outline how you can create an image for your edge runtime using the wam/wpm cli tools provided with webMethods.
Download the project scaffold and other resources for your webMethods.io project
In the article above, Develop anywhere - A practical guide to using packages with webMethods.io we explained how you can externalize the source code for your webMethods.io project, this also includes the scaffold file. This file describes all of the assets and configuration required for your project to function and it is the basis for generating the necessary edge runtime images.
Download my example webMethods Integration project from the following git repo and clone it to your local computer so that you can extract the scaffold file that we use describe the integration project e.g.
$ git clone https://github.com/johnpcarter/TechCommunityDemoProject.git
Once cloned you can find the project scaffold in the ../config/scaffolding folder.
Configure wam/wpm for local usage
wpm/wam are the command line tools to allow you to build your container images easily. I have bundled them with the project you just downloaded and you can reference them from the above cloned folder. Edit the PATH attribute in your environment to include the wpm/bin folder so that you can run the scripts from wherever you require.
$ setenv PATH=$PATH/<...>/TechCommunityDemoProject/wpm/bin
Check that either wam or wpm is executable e.g.
$ wam -h
webMethods Application Manager (wam)
version 0.0.3
Use this tool to generate a Dockerfile to build a webMethods runtime image from a project scaffold(s) or edge manifest file
usage: wam
-b,--base-image <arg> Base image, defaults to sagcr.azurecr.io/webmethods-edge-runtime:latest
-d,--dockerfile-name <arg> Name of Dockerfile to generate, defaults to Dockerfile
-e,--edge-manifest <arg> Edge manifest file to read
-h,--help Help information
-m,--edge-manifest-name <arg> Write an intermediate edge manifest with the given name instead of a Dockerfile
-p,--projects <arg> Project scaffold file or directory containing project scaffolds
-s,--service <arg> The project service to reference, if omitted all services will be included
-t,--target-dir <arg> Target directory for installation, defaults to /opt/softwareag/IntegrationServer
-v,--version Get the version number of this cli tool
You may need to set the execute permission if the above command fails or check the JAVA_HOME directory in the setenv.sh script, found in the …/bin folder.
Getting the required git/wpm tokens
In order to build an image we will use the wam cli tool to generate a docker file. The generated Dockerfile will reference the wpm cli tool that is baked into our base image, to pull the packages, and if any of these packages are secured then an access token will need to be included. You can add these tokens either to the generated Dockerfile or update the project scaffold beforehand.
To add the git or wpm acces tokens open the project scaffold file and scroll down to the section packages and replace the gitUsername and ADD gitToken entries with your own id and token. For wpm references such as the WmJDBCAdapter you will need to login to https://packages.webmethods.io and generate an access from the settings page.
...
...
packages:
- name: TechCommunityDemoProject
gitServerName: JohnBoy
gitUrl: https://github.com/johnpcarter
gitUsername: johnpcarter
gitToken: <INSERT YOUR GIT TOKEN HERE>
gitBranch: main
- name: JcPublicTools
version: "5.0"
gitServerName: JohnBoy
gitUrl: https://github.com/johnpcarter
gitUsername: johnpcarter
gitToken: <INSERT YOUR GIT TOKEN HERE>
gitTag: v2.1.0
- name: WmJDBCAdapter
wpmServer: https://packages.webmethods.io
wpmRegistry: licensed
wpmToken: <INSERT YOUR PACKAGE REGISTRY ACCESS TOKEN HERE>
.
Generate an image for the entire project
Now we can generate a Dockerfile to build our image, simply cd into you work directory where you cloned the project package and run the following command
$ cd .
$ wam -p TechCommunityDemoProject/config/scaffolding
webMethods Application Manager (wam)
version 0.0.3
extracting all services assets from project scaffold /Users/jcart/Documents/Work/wpm/EdgeJcHelloWorld/config/scaffolding/EdgeJcHelloWorld.yml
Generating docker file Dockerfile
Generating application.properties file
As you can see it has generated a Dockerfile and an application.properties file. If you open the Dockerfile you will see something like the following (albeit I have removed the tokens that you will have included above).
FROM sagcr.azurecr.io/webmethods-edge-runtime:latest
WORKDIR /opt/softwareag/wpm
RUN wpm install -ws https://packages.softwareag.com -wr softwareag -j eyJhb... -d /opt/softwareag/IntegrationServer WmJDBCAdapter
RUN wpm install -u johnpcarter -p ghp_... -r https://github.com/johnpcarter -d /opt/softwareag/IntegrationServer EdgeJcHelloWorld
RUN wpm install -u johnpcarter -p ghp_... -r https://github.com/johnpcarter -d /opt/softwareag/IntegrationServer JcPublicTools
WORKDIR /
COPY application.properties /opt/softwareag/IntegrationServer/application.properties
You can see that it is leveraging the baked-in wpm command to ensure that packages can be easily pulled into the image.
Using e2e tracing
If you want to trace the execution of services running in your edge runtime in our end 2 end monitoring solution then add the following lines to your application.properties file before building.
settings.watt.server.audit.service.include=$toplevel,edge.*,project.*
settings.watt.server.audit.service.exclude=$wmservices
Build your image
Before you run the docker build command replace the latest tag with the version that you want to use, current version as of publishing this article is 11.0.10. Now you can run the docker build command e.g.
$ docker build -t my-edge-runtime:1.0 .
[+] Building 0.4s (11/11) FINISHED docker:default
=> [internal] load build definition from Dockerfile 0.0s
=> => transferring dockerfile: 1.32kB 0.0s
=> [internal] load .dockerignore 0.0s
=> => transferring context: 2B 0.0s
=> [internal] load metadata for sagcr.azurecr.io/webmethods-edge-runtime:11.0.10 0.0s
=> [1/7] FROM sagcr.azurecr.io/webmethods-edge-runtime:11.0.10 0.0s
=> [internal] load build context 0.0s
=> => transferring context: 3.70kB 0.0s
=> CACHED [2/7] WORKDIR /opt/softwareag/wpm 0.0s
=> CACHED [3/7] RUN /opt/softwareag/wpm/bin/wpm.sh install -u johnpcarter -p ghp_... -r https://github.com/johnpcarter 0.0s
=> CACHED [4/7] RUN /opt/softwareag/wpm/bin/wpm.sh install -ws https://packages.softwareag.com -wr softwareag -j eyJhb... 0.0s
=> CACHED [5/7] RUN /opt/softwareag/wpm/bin/wpm.sh install -u johnpcarter -p ghp_... -r https://github.com/johnpcarter 0.0s
=> [6/7] COPY application.properties /opt/softwareag/IntegrationServer/application.properties 0.2s
=> exporting to image 0.1s
=> => exporting layers 0.1s
=> => writing image sha256:d1046c96c7c818764dc25fbf846b4ec52d014d41087aae3899201044ac91036d 0.0s
=> => naming to docker.io/library/my-edge-runtime:latest
Spinning up the container
Starting the container is identical to the pairing process as as shown when registering or starting the container from the control plane in webMethods.io, you only need to replace the image with that of your own above, and with one other small change.
docker run -p 5555:5555 -d -e SAG_IS_CLOUD_ALLOWSYNC=false -e SAG_IS_CLOUD_REGISTER_URL=https://wmintanywhere.int-aw-au.webmethods.io -e SAG_IS_EDGE_CLOUD_ALIAS=EdgeRuntime_JC_MAC -e SAG_IS_CLOUD_REGISTER_TOKEN=... --name=JC_MAC_1bd35694 my-edge-runtime:latest
You will perhaps notice that I have also added an extra environment variable SAG_IS_CLOUD_ALLOWSYNC=false, this is to ensure that the container does attempt or accept any updates to its packages. The whole point of basing our runtime on a prebuilt image is to avoid the container getting updated so this is an important change to make.
Slicing up your project into separate edge runtimes and merging projects.
In some circumstances you may only want to partially deploy your project to a runtime or even deploy several project fragments to the same runtime, wam allows you to do both;
To generate a fragment simply specify the name of the project service that you plan to deploy to the runtime from the project using the -s --service switch. In which case it will only deploy the dependent packages as required by the service along with the project package. It will also ensure that only connections that are required by that service or dependent services will be activated, all other connections will be disabled by default.
You can also combine projects by simply running the map command several times, whilst using the option -m --edge-manifest-name to generate an edge manifest file rather than directly a Dockerfile. In this case, it will update the existing file if it already exists with the new assets. The command will generate warnings if it detects conflicts when merging projects such as conflicting package versions.
Next steps
The above startup is yet again a very manual step and would not be a valid scenario for spinning up your container in a production environment. In my next article, I will show you how you now take advantage of your pre-built image to configure a Kubernetes deployment to auto-deploy and scale a number of pods for your project.
Useful links | Relevant resources
Develop anywhere - A practical guide to using packages with webMethods.io
Building an integration microservice from A to Z with webMethods and Kubernetes
wpm.zip cli to download
#tutorial
#webMethods
#Integration-Server-and-ESB
#Super-iPaaS