Configuring Alerts in Cloud Logs using Event Notification channel

In this tutorial, you learn how to set up alert notifications in IBM Cloud Logs with IBM Cloud Event Notifications.

Before You Begin

Ensure prerequisites like an IBM Cloud account with necessary permissions are met. Your IBM Cloud Log Monitoring instance and IBM Cloud Event Notifications instances must be in the same region to communicate with one another.

Step 1: Log in to IBM Cloud

1. Log in to your IBM Cloud account.
2. Click IBM Cloud dashboard to launch the IBM Cloud dashboard.
3. Log in with your user ID and password. The IBM Cloud Dashboard opens.

Step 2: Create an IBM Cloud Event Notifications Service Instance

1. Click Catalog.
2. In Search the catalog field, enter Event Notifications.
3. In the Create tab, enter:
• For Select a location, select Dallas (us-south).
• For Select a pricing plan, select Lite.
• For Service name, enter a name of your choice. For example, enter my_event_notifications.
• For Select a resource group, enter your resource group.
4. Read and agree to the license agreements.
5. Click Create. Your IBM Cloud Event Notifications instance is created and displayed.

Step 3: Configure Permissions

To grant an instance of the IBM Cloud Monitoring service access to an instance of the IBM Cloud Event Notifications service, follow these steps:

1. Click Manage > Access (IAM).
2. Click Authorizations.
3. Click Create.
4. Set the following:
• For Source service, select IBM Cloud Monitoring.
• For How do you want to scope the access?, select All resources.
• For Target service, select Event Notifications.
• For How do you want to scope the access?, select All resources.
• For Service access, make sure Reader and Event Source Manager are selected.
5. Click Authorize.

For more information on how to define authorizations, see Using authorizations to grant access between services.

Step 4: Configure a Notification Source in IBM Cloud Event Notifications

1. Click the menu icon in the IBM Cloud Console.
2. In the navigation menu, select Observability > Monitoring.
3. Select an instance in the us-south region, then click Open Dashboard. The Web UI opens.
4. In the navigation menu, click the circle icon with your initials.
5. Click Settings > Notification Channels > Add Notification Channel.
6. Select IBM Event Notifications.
7. For Event Notifications Instance, select the IBM Cloud Event Notifications you created.
8. For Channel Name, give your channel a unique name. For example, my_event_notification_channel.
9. Leave all options enabled and the Shared With team as the default value.
10. Click Save.

11.  Click the menu icon > Resource list.

12.  Open Services and software.

13.  Open the IBM Cloud Event Notifications instance you created. For example, my_event_notifications.

14.  Click Sources.

When you configure the channel in Monitoring, a source, with the same name as your Monitoring instance name, is automatically added to your IBM Cloud Event Notifications Sources list.

Step 5: Create an IBM Cloud Event Notifications Topic

1. Click Topics > Create.
2. Enter the following details:
• Name: For example, MyMonitoringTopic.
• Source: Select the IBM Cloud Event Notifications source named after your IBM Cloud Monitoring instance.
• Event Type: Select Alert.
• Event Subtype: Select Metric.
• Severity: Select Info Severity.

3. Click Add a condition.
4. Click Create. Your topic will be displayed in the Topics list.

Step 6: Create an IBM Cloud Event Notifications Destination

1. Click Destinations.
2. Notice that, by default, an IBM Cloud Email service is defined. No further configuration is needed.

Step 7: Create an IBM Cloud Event Notifications Email Subscription

1. Click Subscriptions > Create.
2. In the subscription panel, enter:
• Name: For example, MyMonitoringSubscription.
• Topic: Select the topic you created, such as MyMonitoringTopic.
• Destination: Select IBM Cloud Email service.
• Recipients: Enter a valid email address, for example, MyEmail@MyCompany.com.

3. Click Create. Your subscription will be added to the Subscriptions list.

Step 8: Create an Outbound Integration for Cloud Logging Instance

1. Click the Navigation Menu icon > Resource list.
2. Select your instance of IBM Cloud Logs.
3. In the navigation, click Integrations > Outbound integrations.
4. Find Event Notifications and click Add.
5. Enter a name for the integration.
6. Select the Event Notifications service instance you want to connect to your IBM Cloud Logs instance.
7. Select the Endpoint Type (public or private). For more information, see Service endpoints for Event Notifications
8. Click Save.

Step 9: Create an Alert in IBM Cloud Logs

9A: Launch Alerts Management

1. Click the Navigation Menu icon > Resource list.
2. Select your instance of IBM Cloud Logs.
3. Click Alerts > Alerts Management > New alert.

9B: Choose the Type of Alert

1. Enter a name and description.
• Name and description can have a maximum of 4096 characters.
2. Define the Severity (Info, Warning, Error, or Critical). For this tutorial, select Info.
3. Add Labels. Labels are key:value pairs that you can use later for quick searching.
4. Choose the alert type. For more information, see Alert types.

9C: Specify Logs for Filtering

1. Define queries for logs, for example:
• To trigger an alert when POST requests with a 403 response are identified, enter POST 403.

9D: Specify Triggering Conditions

1. Select an evaluation type:
• Notify Immediately: Notified immediately, as soon as 1 log record is evaluated to match the filtering criteria.
• Less Than: Notified when the count of the entries matching the alert definition is less than the chosen threshold.
• More Than: Notified when the count of the entries matching the alert definition is more than the chosen threshold.
• More Than Usual: Automatically detects anomalies.
• Rolling Window: Defines a fixed period of time such as 5 minutes, regardless of matching data and any alerts triggered as a result of the query
• Dynamic Duration: Window changes the queried period when data matching the query triggers an alert.

9E: Configure Notification Details

1. Set Notify every to define alert frequency (default: 10 minutes).
2. Enable Notify when resolved to receive updates when the condition is resolved.

When the alert's condition is no longer triggering events, the event that is trigered initially is marked as resolved.

3. Enable Enable phantom mode to indicate that this alert is a phantom alert
4. Add an integration

9F: Schedule Alerts and Include Log Content

1. Set schedules for specific days and times.
2. Choose whether to include full log text or specific fields.

9G: Save the Alert

1. Verify the alert

Click Verify to evaluate data to find out how many times the alert matched the criteria in the last 24 hours.

Important: Verify evaluates data in the Priority insights pipeline only. If your alert is configured to trigger on data that is available in the Analyze and alert pipeline, notice that this feature is not available.

2. Click CREATE ALERT.