Code Engine provides a Kubernetes based serverless platform for applications, jobs, and functions, which can scale down to zero when there is no work to do. This pattern is attractive to solution designers as it can result in significant cost benefit in processing intermittent or variable workloads. For web applications it is relatively easy to instatiate and scale-up containers in response to incoming http requests. For enterprise messaging, application containers need to scale in response to messages on queues. This is where the new IBM MQ observer sample comes in, allowing applications to register their interest in IBM MQ queues and receiving ‘wake-up’ notifications when messages are available.
The IBM MQ Observer falls into line with the new IBM Code Engine strategy for notifying application containers. Why new? Thehad multi-tenancy watchers, which would grab the event, then wake up and notify your applications. Eg. Subscribing to Object Storage events. This has some drawbacks. You must hand your keys and data to the watcher, which could use sophisticated APIs with high quality of service to capture the event and data. Once captured the watcher then has the burden of responsibility to deliver the event data as a notification to your scalable application. This design could lead to an in-flight edge case where the event had been successfully captured, but is subsequently lost before it could be delivered successfully.
The new model has single-tenancy observers which you run. The observers listen for events. The observers don’t capture the event, instead they wake up applications or jobs letting them use the high quality of service APIs to process the events. If the event notification between observer and job fails, then there will be a fresh notification in the next observation cycle. Samples in the Code Engine repository can get you started.
A few years a go we created the tutorial Write and run serverless MQ applications in IBM Cloud that provided sample code that allowed you to run a single tenancy observer that captured a message ID and sent that as a notification to messaging applications. We allowed the observer to run anywhere, you have the choice as to whether it runs as a Code Engine application or elsewhere. We will be updating the tutorial to use the new IBM MQ observer sample.
The new IBM MQ observer sample provides new features and improvements over our previous observer sample design. For example:
- The observer can be given a seed set of registrations on start-up should the messaging jobs already exist. This avoids the need to open ports and run unnecessary http server instances.
- The observer allows for dynamic registrations. i.e., I have an observer running, and a new messaging job that I want to register for notifications.
- Applications can revoke their registrations.
- The observer can monitor multiple queues on multiple queue managers.
- The observer can also be hosted in Code Engine, but can also run in an external runtime environment.
- Code Engine Applications can register http URL endpoints.
- Code Engine Jobs can register as job names.
- Registration data may be ephemeral, but support for a persistenta persistent store is included.
- Cycle times, e.g., the time duration between queue depth queries and the time duration between persisting data are customisable through environment variables, and configmaps.
- The observer can be made to back-off for a duration sending notifications to applications and jobs that fail to start.
- The observer uses the MQ REST admin API to monitor queue depths.
- Application and job developers can choose how to process messages.
- A sample consumer is provided, that uses the IBM MQ C based MQI API.
- Both the observer and message consumer saamples are implemented in Go.
- A ‘deploy script’ is provided that creates secrets, configmaps and deploys the observer and sample consumer as a Code Engine job definition.