Integration facilitates the exchange of information between systems and may be realised through the use of technologies such as messaging, API gateways and the associated protocols. In this article, we're going to use IBM Event Notifications as the integration technology. We'll illustrate through code and examples on how two sample applications may be integrated asynchronously.
Integrating two systems using a messaging infrastructure involves establishing a communication channel between the systems, enabling them to exchange data and coordinate actions in real-time. This approach offers several benefits, including increased efficiency, reduced complexity, and improved fault tolerance.
To integrate two systems using messaging, we need to follow these general steps:
- Choose a messaging channel. Both of our systems under consideration are RESTful systems. In this case, Webhooks can be used for event-driven messaging over IBM Event Notifications. Webhooks are a popular method for integrating applications and services, allowing them to communicate directly with each other when specific events occur.
- Set up a messaging broker. We'll use IBM Event Notifications to integrate two systems. It is a fully managed, serverless event bus service that allows you to create, manage, and monitor event-driven architectures.
- Define a message structure. To ensure seamless communication between the systems, messages in JSON format will be used to define payloads to exchange information.
- Implement message producers and consumers. A web service at each end will be created to invoke and receive Webhooks.
- Test the integration. We'll use open-source tools available to develop and test the integration.
Context Diagram
There are two applications - alerting application that detects an alert and wants to send the alert to another application that may be aggregating alerts from many applications, create an incident to investigate alerts and lets the alerting application know about the result of the investigation. Here, we're illustrating only one way communication from alerting application to alert aggregator. However, without loss of generality, the same approach can be used for bi-directional information exchange. An admin will provision the necessary infrastructure on the IBM Cloud to facilitate the integration.
Terraform is an open-source infrastructure as code (IaC) tool that enables users to define and provision cloud resources in a declarative manner. We'll use Terraform to declare Event Notifications resources for deployment.
Pre-requisites
- IBM IAM API Key
- IBM Event Notifications instance id
- Webhook URL to receive event payload
Before we can use Terraform, download and install Terraform. Create an IBM Cloud API Key in IAM to be used to create and use Event Notifications resources - both from Terraform and from applications.
Building Infrastructure-as-Code
Before we declare the resources that Terraform will create for us, let us look at the resources requires to work with IBM Event Notifications.
To start with, we need following resources to be created on an IBM Event Notifications instance:
- Source
- Topic
- Destination
- Subscription
Create a directory to keep all the Terraform configuration files.
versions.tf
versions.tf downloads the required providers and their versions.
terraform {
required_version = ">=1.0.0, <2.0"
required_providers {
ibm = {
source = "IBM-Cloud/ibm"
}
}
}
provider.tf
provider.tf creates the required providers.
variable "ibmcloud_api_key" {}
variable "region" {}
provider "ibm" {
ibmcloud_api_key = var.ibmcloud_api_key
region = var.region
}
variables.tf
variables.tf stores values that are required for Terraform at run time.
variable "instance_id" {
type = string
default = "<instance id>"
description = "Event Notifications service instanceId."
}
variable "destination_name" {
type = string
default = "test"
description = "Test"
}
variable "destination_description" {
type = string
default = "Destination for Test"
description = "Description for the Test."
}
variable "webhook_type" {
default = "webhook"
type = string
description = "Type of the destination"
}
variable "webhook_verb" {
default = "POST"
type = string
description = "Type of verb for the destination"
}
variable "webhook_URL" {
type = string
description = "URL of the webhook"
default = "<webhook-url>"
}
terraform.tfvars
terraform.tfvars stores API Key that are required for Terraform at run time. This file should not be committed to Github repository.
ibmcloud_api_key = "<IBM Cloud API Key>"
region = "us-south"
destinations.tf
destinations.tf uses Event Notifications to create destinations.
resource "ibm_en_destination_webhook" "destination1" {
type = var.webhook_type
name = var.destination_name
description = var.destination_description
instance_guid = var.instance_id
config {
params {
url = var.webhook_URL
verb = var.webhook_verb
}
}
}
topics.tf
topics.tf uses Event Notifications to create topics.
resource "ibm_en_topic" "topic1" {
instance_guid = var.instance_id
description = "Topic to deliver alerts"
name = "alerts"
}
subscriptions.tf
subscriptions.tf uses Event Notifications to create subscriptions.
resource "ibm_en_subscription_webhook" "subscription1" {
instance_guid = var.instance_id
name = "Test"
description = "Subscribe to Alerts"
topic_id = ibm_en_topic.topic1.topic_id
destination_id = ibm_en_destination_webhook.destination1.destination_id
attributes {
signing_enabled = true
}
}
sources.tf
sources.tf creates event sources that will publish events to destinations.
resource "ibm_en_source" "alert_source" {
instance_guid = var.instance_id
name = "EN Source"
description = "API source for Event Notifications destinations"
enabled = true
}
Configure IBM Event Notifications
- Change directory to terraform where the Terraform resource files have been created.
- In the file
variables.tf, provide all the variable values, especially, instance_id and webhook_URL.
- Initialise Terraform. The Terraform init command prepares the environment by ensuring the directory is properly configured:
% terraform init
Initializing the backend...
Initializing provider plugins...
- Finding latest version of ibm-cloud/ibm...
- Installing ibm-cloud/ibm v1.62.0...
- Installed ibm-cloud/ibm v1.62.0 (self-signed, key ID AAD3B791C49CC253)
Partner and community providers are signed by their developers.
If you'd like to know more about provider signing, you can read about it here:
https://www.terraform.io/docs/cli/plugins/signing.html
Terraform has created a lock file .terraform.lock.hcl to record the provider
selections it made above. Include this file in your version control repository
so that Terraform can guarantee to make the same selections by default when
you run "terraform init" in the future.
Terraform has been successfully initialized!
You may now begin working with Terraform. Try running "terraform plan" to see
any changes that are required for your infrastructure. All Terraform commands
should now work.
If you ever set or change modules or backend configuration for Terraform,
rerun this command to reinitialize your working directory. If you forget, other
commands will detect it and remind you to do so if necessary.
- Run
terraform plan. The Terraform plan command compares the declared resources with the state file to print the resources to be created, altered, or destroyed.
% terraform plan
Terraform used the selected providers to generate the following execution plan. Resource actions are indicated with the following
symbols:
+ create
Terraform will perform the following actions:
# ibm_en_destination_webhook.destination1 will be created
+ resource "ibm_en_destination_webhook" "destination1" {
+ description = "Destination for Test"
+ destination_id = (known after apply)
+ id = (known after apply)
+ instance_guid = "6a49cccc-a693-466b-8c1e-291d7b4ff091"
+ name = "test"
+ subscription_count = (known after apply)
+ subscription_names = (known after apply)
+ type = "webhook"
+ updated_at = (known after apply)
+ config {
+ params {
+ url = "https://smee.io/YykZuCem2lrnLP4F"
+ verb = "POST"
}
}
}
# ibm_en_subscription_webhook.subscription1 will be created
+ resource "ibm_en_subscription_webhook" "subscription1" {
+ description = "Subscribe to Ransomware Alerts from Storage Insights"
+ destination_id = "6a49cccc-a693-466b-8c1e-291d7b4ff091"
+ destination_name = (known after apply)
+ destination_type = (known after apply)
+ id = (known after apply)
+ instance_guid = "6a49cccc-a693-466b-8c1e-291d7b4ff091"
+ name = "IBMDefender"
+ subscription_id = (known after apply)
+ topic_id = (known after apply)
+ topic_name = (known after apply)
+ updated_at = (known after apply)
+ attributes {
+ signing_enabled = true
}
}
# ibm_en_topic.topic1 will be created
+ resource "ibm_en_topic" "topic1" {
+ description = "Topic to deliver ransomware alerts"
+ id = (known after apply)
+ instance_guid = "6a49cccc-a693-466b-8c1e-291d7b4ff091"
+ name = "ransomware-alerts"
+ source_count = (known after apply)
+ subscription_count = (known after apply)
+ subscriptions = (known after apply)
+ topic_id = (known after apply)
+ updated_at = (known after apply)
}
# ibm_en_source.en_source will be created
+ resource "ibm_en_source" "en_source" {
+ description = "API source for Event Notifications destinations"
+ enabled = true
+ id = (known after apply)
+ instance_guid = "6a49cccc-a693-466b-8c1e-291d7b4ff091"
+ name = "EN Source"
+ source_id = (known after apply)
+ updated_at = (known after apply)
}
Plan: 4 to add, 0 to change, 0 to destroy.
─────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────
- Run
terraform apply. The Terraform apply command implements the changes that are declared during the plan step and deploys the resource to the IBM Cloud.
% terraform apply -auto-approve
Terraform used the selected providers to generate the following execution plan. Resource actions are indicated with the following
symbols:
+ create
Terraform will perform the following actions:
# ibm_en_destination_webhook.destination1 will be created
+ resource "ibm_en_destination_webhook" "destination1" {
+ description = "Destination for Test"
+ destination_id = (known after apply)
+ id = (known after apply)
+ instance_guid = "6a49cccc-a693-466b-8c1e-291d7b4ff091"
+ name = "test"
+ subscription_count = (known after apply)
+ subscription_names = (known after apply)
+ type = "webhook"
+ updated_at = (known after apply)
+ config {
+ params {
+ url = "https://smee.io/YykZuCem2lrnLP4F"
+ verb = "POST"
}
}
}
# ibm_en_subscription_webhook.subscription1 will be created
+ resource "ibm_en_subscription_webhook" "subscription1" {
+ description = "Subscribe to Ransomware Alerts from Storage Insights"
+ destination_id = "6a49cccc-a693-466b-8c1e-291d7b4ff091"
+ destination_name = (known after apply)
+ destination_type = (known after apply)
+ id = (known after apply)
+ instance_guid = "6a49cccc-a693-466b-8c1e-291d7b4ff091"
+ name = "IBMDefender"
+ subscription_id = (known after apply)
+ topic_id = (known after apply)
+ topic_name = (known after apply)
+ updated_at = (known after apply)
+ attributes {
+ signing_enabled = true
}
}
# ibm_en_topic.topic1 will be created
+ resource "ibm_en_topic" "topic1" {
+ description = "Topic to deliver ransomware alerts"
+ id = (known after apply)
+ instance_guid = "6a49cccc-a693-466b-8c1e-291d7b4ff091"
+ name = "ransomware-alerts"
+ source_count = (known after apply)
+ subscription_count = (known after apply)
+ subscriptions = (known after apply)
+ topic_id = (known after apply)
+ updated_at = (known after apply)
}
# ibm_en_source.en_source will be created
+ resource "ibm_en_source" "en_source" {
+ description = "API source for Event Notifications destinations"
+ enabled = true
+ id = (known after apply)
+ instance_guid = "6a49cccc-a693-466b-8c1e-291d7b4ff091"
+ name = "EN Source"
+ source_id = (known after apply)
+ updated_at = (known after apply)
}
Plan: 4 to add, 0 to change, 0 to destroy.
...
Apply complete! Resources: 4 added, 0 changed, 0 destroyed.
- Validate resources within the IBM Cloud by running
terraform show.
% terraform show
# ibm_en_destination_webhook.destination1:
resource "ibm_en_destination_webhook" "destination1" {
description = "Destination for Test"
destination_id = "630190f5-f604-4f78-a343-e8fb00cbc5e1"
id = "6a49cccc-a693-466b-8c1e-291d7b4ff091/630190f5-f604-4f78-a343-e8fb00cbc5e1"
instance_guid = "6a49cccc-a693-466b-8c1e-291d7b4ff091"
name = "test"
subscription_count = 0
subscription_names = []
type = "webhook"
updated_at = "2024-03-04T10:01:32.334Z"
config {
params {
custom_headers = {}
sensitive_headers = []
url = "https://smee.io/YykZuCem2lrnLP4F"
verb = "POST"
}
}
}
# ibm_en_source.en_source:
resource "ibm_en_source" "en_source" {
description = "API source for Event Notifications destinations"
enabled = true
id = "6a49cccc-a693-466b-8c1e-291d7b4ff091/3e9984ba-f09e-491d-b745-93167b0e3f26:api"
instance_guid = "6a49cccc-a693-466b-8c1e-291d7b4ff091"
name = "EN Source"
source_id = "3e9984ba-f09e-491d-b745-93167b0e3f26:api"
updated_at = "2024-03-04T10:29:11.758Z"
}
# ibm_en_subscription_webhook.subscription1:
resource "ibm_en_subscription_webhook" "subscription1" {
description = "Subscribe to Alerts"
destination_id = "630190f5-f604-4f78-a343-e8fb00cbc5e1"
destination_name = "test"
destination_type = "webhook"
id = "6a49cccc-a693-466b-8c1e-291d7b4ff091/05f563f0-6ce4-47d7-8070-2cc514e445ee"
instance_guid = "6a49cccc-a693-466b-8c1e-291d7b4ff091"
name = "Test"
subscription_id = "05f563f0-6ce4-47d7-8070-2cc514e445ee"
topic_id = "be7d95ad-295b-402f-aa0f-4338ce0de94b"
topic_name = "alerts"
updated_at = "2024-03-04T10:13:33.725105Z"
attributes {
signing_enabled = true
}
}
# ibm_en_topic.topic1:
resource "ibm_en_topic" "topic1" {
description = "Topic to deliver alerts"
id = "6a49cccc-a693-466b-8c1e-291d7b4ff091/be7d95ad-295b-402f-aa0f-4338ce0de94b"
instance_guid = "6a49cccc-a693-466b-8c1e-291d7b4ff091"
name = "alerts"
source_count = 0
subscription_count = 1
subscriptions = []
topic_id = "be7d95ad-295b-402f-aa0f-4338ce0de94b"
updated_at = "2024-03-04T10:08:26.676862Z"
}
Event Publisher
Now that we have the necessary resources created by Terraform, next we will use the Event Notifications SDK to publish events to the instance. Following is the Java code to create and send notification.
private static void sendIBMEventNotification(JsonObject payload, String enInstanceId, String enSourceId) {
String sourceMethod = "sendIBMEventNotification";
HashMap<String, Object> eventMap = new Gson().fromJson(payload.toString(), HashMap.class);
try {
NotificationCreate body = new NotificationCreate.Builder()
.ibmenseverity("HIGH")
.id(payload.getString("id"))
.source("IBM Storage Insights")
.ibmensourceid(enSourceId)
.type("Alert Notification")
.time(new java.util.Date())
.data(eventMap)
.datacontenttype("application/json")
.ibmensubject("Alert")
.specversion("1.0")
.ibmendefaultshort("Alert")
.ibmendefaultlong("Alert from Storage Insights")
.subject("Alert Aggregator")
.build();
SendNotificationsOptions sendNotificationsOptions = new SendNotificationsOptions.Builder()
.instanceId(enInstanceId)
.body(body)
.build();
Response<NotificationResponse> response = eventNotificationsService.sendNotifications(sendNotificationsOptions).execute();
if (response != null) {
logger.info("IBM Event Notification sent successfully, response: " + response.toString());
} else {
logger.error("Response is null");
throw new RuntimeException("Response is null");
}
NotificationResponse notificationResponse = response.getResult();
if (notificationResponse != null) {
logger.info("IBM Event Notification sent successfully, result: " + notificationResponse.toString());
}
} catch (ServiceResponseException e) {
logger.error(String.format("Service returned status code %s: %s%nError details: %s", e.getStatusCode(), e.getMessage(), e.getDebuggingInfo()), e);
throw new RuntimeException("Error sending IBM event notification ", e);
}
}
Handling webhook deliveries
When the event publisher publishes an event to the IBM Event Notifications, it is configured to invoke a Webhook that was declared while provisioning resources. To test Webhook deliveries, we can use
https://smee.io to create a Webhook URL and use the same in the provisioning step.
The event payload will be posted to the Webhook URL as shown below.
This completes the flow of event from the publisher to the receiver via the IBM Event Notifications.
In this article, we set out to integrate two systems over messaging infrastructure provided by IBM Event Notifications. We provisioned the required resources declaratively using Terraform. We, then, used Event Notifications Java SDK to create a publisher that published events to the IBM Event Notifications. Event Notifications invoked a Webhook configured as the destination. We used a tool named smee.io to receive the Webhook payload and display them.
The code is available on Github.