Alan Chatt, Dale Lane
Last month, we announced the first iteration of Event Endpoint Management - a new and exciting capability in Cloud Pak for Integration. Extending API management concepts into the asynchronous domain of events is is part of the broader work we’re doing to enable Multi-Form API Management.
In our last post, we discussed the need for event endpoint management. In this post, we want to describe what this should include, by expanding on how API management concepts apply to events.
We’ll do this using the 4 D’s of Event Endpoint Management:
Events need to be described in a way that provides enough information for another developer to make use of them.
The description obviously needs to include the format or schema of data contained within the event. It also needs to contain information about the topic the event resides on, connection details for the broker hosting the topic, and the protocol to use to access the events.
In addition to technical information, it’s often equally useful to have a description of what the events represent, and contact details for who owns the systems that emit these events. This meta information makes it easier for application developers to identify appropriate situations to use these events, and how to quickly find more details if required.
The standard we use to describe APIs is OpenAPI. For events, we’re using AsyncAPI. This is a newer standard, which recently joined the Linux Foundation. Its similarity to OpenAPI and the growing adoption make it an ideal way to describe Event Endpoints.
Generating events is only useful if other people make use of them. For developers to make use of them they must be discoverable.
API management gives us the concept of a developer portal for listing all of the APIs available in an organised and searchable way. This concept directly applies to event endpoints as well.
Furthermore, we believe this should be one catalog for all interfaces, including both APIs and events. Where a system offers multiple interfaces (both APIs and events) and allows developers to pick the right form for their use case, it makes sense to group these together.
One important aspect of API management is how developers are able to self-serve access to APIs that they discover in the portal. Enabling this simple interaction probably does more to promote agility than anything else because there is no faster way to learn about something than to try it out.
Enabling self-service access to APIs requires many protections to be specified and enforced, such as access policies (who is eligible to access this API), protection for the backend implementation (usage limits), and so on.
It also requires usage monitoring and analytics to record who is doing what - both to spot anomalous behaviour, and to provide insights for how APIs can be improved.
The same “self-service” access makes equal sense for events: to enable developers to have quick access to these additional event-driven ways of interacting with a system. This self-service access will require similar protections. Some will be the same, such as access control policies. Some will be analogous, such as limiting connections or subscriptions rather than API calls. Some will be entirely specific to the event-driven model or protocol, such as preventing inadvertent collisions between separate Kafka consumers by preventing them from using the same consumer group.
The owner of an event endpoint will need a way to specify these controls and protections, and we will need facilities that can enforce them.
APIs provide an abstracted interface, independent from how a system is implemented. As long as we are able to keep the API specification consistent, we’re free to change the implementation of the caller or the system behind the API independently, effectively decoupling their lifecycles.
Event endpoints can help us achieve the same, but there are additional considerations in the event domain, partly due to a broader range of protocols in use, and partly due to the need for a message broker.
In order to truly enable decoupling for event endpoints, we will need to consider how we might translate between different protocols (both where events are created, and also how events need to be consumed), with IBM MQ, AMQP (1.0 and 0.91 being widely used), Apache Kafka, MQTT, webhooks, WebSockets, and so on all being examples of protocols in wide use.
We also need to consider how we decouple producers and consumers operationally: how do we prevent consumers from being able to create operational problems for the event endpoint provider. Some of this can be addressed by the protections described above, but each protocol and messaging system will have their own considerations.
Event Endpoint Management
In summary, we have more to do to reach the level of management maturity for event endpoints that we have for APIs, but we also have over a decade of experience with APIs to draw upon.
The first step for Event Endpoint Management, released last month, focuses on Description and Discovery. We’d love to hear people’s experiences with this. Our goal is to elevate events to being a first-class interface that developers can use alongside APIs for any applications or systems they want to interact with.
If you'd like to hear more about what we’re doing, please join our webinar, "Automate your integration - What's new in Cloud Pak for Integration 2021.1.1" on April 15th.