In this post, I want to share examples of how watsonx can enhance the event processing flows you create using IBM Event Processing.
I'll describe how Event Processing and watsonx complement each other, and share a couple of simple examples of what this looks like in action.
Enhancing Event Processing with watsonx
Event Processing is a low-code tool for creating event processing flows. Based on Apache Flink, Event Processing makes it easy to process streams of events in a stateful way.
Windows let you aggregate - processing events in the context of other events that have happened before or after. Joins let you correlate - processing events in the context of other streams of events.
Generative AI such as watsonx can be used to take your event processing flows a step further. Once you've used Flink to aggregate and correlate multiple raw streams of events, you can use watsonx to Summarize and explain the insights you find.
In this post, I'll share two very simple examples to show how easy it is to get started with this.
I'm focusing on summarization in these examples, but that's just one way that you can use machine learning models with your event processing projects.
Other examples include:
- clustering - identify groups of related events
- question answering - orchestrate responding to questions
- classification - recognise and categorise events
- extraction - identify key information within events
- and many more
The principle for such projects are similar to what I'll show below: use the core capabilities of Event Processing to transform, correlate and aggregate multiple raw streams of events, and enhance this using machine learning to intelligently recognise and respond to the output.
Summarizing stock market activity
Given a stream of stock market activity, generate an easy-to-understand summary of what has happened every hour.
Input:
A Kafka topic with an event every minute detailing stock trade activity over the last minute.
{
"volume": 111052,
"open": 168.98,
"high": 169.11,
"low": 168.96,
"close": 169.07,
"datetime": "2024-05-17 15:59:00",
"stock": "IBM"
}
{
"volume": 32374,
"open": 168.964,
"high": 168.99,
"low": 168.95,
"close": 168.975,
"datetime": "2024-05-17 15:58:00",
"stock": "IBM"
}
{
"volume" : 23239,
"open" : 168.97,
"high" : 168.99,
"low" : 168.93,
"close" : 168.96,
"datetime" : "2024-05-17 15:57:00",
"stock" : "IBM"
}
{
"volume": 23358,
"open": 168.89,
"high": 169,
"low": 168.88,
"close": 168.97,
"datetime": "2024-05-17 15:56:00",
"stock": "IBM"
}
{
"volume": 22083,
"open": 168.9,
"high": 168.94,
"low": 168.87,
"close": 168.87,
"datetime": "2024-05-17 15:55:00",
"stock": "IBM"
}
{
"volume": 23331,
"open": 168.93,
"high": 168.98,
"low": 168.89,
"close": 168.91,
"datetime": "2024-05-17 15:54:00",
"stock": "IBM"
}
...
{
"volume": 3998,
"open": 168.46,
"high": 168.46,
"low": 168.4,
"close": 168.46,
"datetime": "2024-05-17 15:01:00",
"stock": "IBM"
}
{
"volume": 2617,
"open": 168.49,
"high": 168.49,
"low": 168.44,
"close": 168.46,
"datetime": "2024-05-17 15:00:00",
"stock": "IBM"
}
Use Event Processing to...
Aggregate the stock trade activity in hourly windows, generating statistical summaries of the activity in this time.
{
"total volume" : 523637,
"mean price" : 168.70,
"low price" : 168.40,
"high price" : 169.11,
"opening price" : 168.49,
"closing price" : 169.07,
"price standard deviation (population)" : 0.145,
"price standard deviation (sample)" : 0.146,
"price variance (population)" : 0.021,
"price variance (sample)" : 0.021,
"start" : "2024-05-17 15:00:00",
"end" : "2024-05-17 16:00:00",
"stock" : "IBM"
}
Use watsonx to...
Summarize the numerical aggregate values in a simple English explanation.
From this data, we can observe some key aspects of trading for a single tech stock over a one-hour period. First, there were a significant number of trades, with 523637 stocks being sold during this time. This indicates a high level of activity in the market.
The stock's price fluctuated throughout the hour, starting at 168.49 and reaching a peak of 169.11 before closing at 169.07. This shows that the stock experienced both buying and selling pressure throughout the hour.
The mean price of the stock during the hour was 168.70, which suggests that the overall trend was slightly upward. However, the stock's price did drop below the mean during the hour, reaching a low of 168.40.
The population variance and sample variance are measures of how spread out the prices are from the mean. In this case, the population variance was 0.021 and the sample variance was also 0.021, indicating that the prices were relatively close to the mean.
The population standard deviation and sample standard deviation give us a sense of the typical distance between the prices and the mean. The population standard deviation was 0.145, while the sample standard deviation was 0.146. This means that the prices were, on average, about 0.146 units away from the mean.
In summary, this data shows that there was a high volume of trading for a single tech stock over a one-hour period, with the stock's price fluctuating around the mean. The prices were relatively close to the mean, with only a small amount of variation from it.
Output:
A Kafka topic with an event every hour containing an easy-to-understand summary of the activity in the last hour.
Demo:
Here is an example of a flow like this in action:
youtu.be/XU-WSLeNIwY
Summarizing weather sensor measurements
Given a stream of measurements from sensors in a weather station, generate a weather report every hour.
Input:
A Kafka topic with an event every two minutes, that contain measurements from a weather station.
{
"datetime": "2024-05-26 08:59:33",
"coord": { "lon": -1.3989, "lat": 51.0267 },
"base": "stations",
"main": {
"temp": 14.15,
"temp_min": 12.99,
"temp_max": 15.45,
"pressure": 1009,
"humidity": 89
},
"visibility": 10000,
"wind": { "speed": 4.63, "deg": 200 },
"clouds": { "all": 75 },
"sys": {
"type": 2,
"id": 2005068,
"country": "GB"
},
"timezone": 3600,
"id": 2652491,
"name": "Compton",
"cod": 200
}
{
"datetime": "2024-05-26 08:57:33",
"coord": { "lon": -1.3989, "lat": 51.0267 },
"base": "stations",
"main": {
"temp": 14.15,
"temp_min": 12.99,
"temp_max": 15.45,
"pressure": 1009,
"humidity": 89
},
"visibility": 10000,
"wind": { "speed": 4.63, "deg": 200 },
"clouds": { "all": 75 },
"sys": {
"type": 2,
"id": 2005068,
"country": "GB"
},
"timezone": 3600,
"id": 2652491,
"name": "Compton",
"cod": 200
}
{
"datetime": "2024-05-26 08:55:33",
"coord": { "lon": -1.3989, "lat": 51.0267 },
"base": "stations",
"main": {
"temp": 14.17,
"temp_min": 12.99,
"temp_max": 16,
"pressure": 1009,
"humidity": 90
},
"visibility": 10000,
"wind": { "speed": 4.63, "deg": 200 },
"clouds": { "all": 75 },
"sys": {
"type": 2,
"id": 2005068,
"country": "GB"
},
"timezone": 3600,
"id": 2652491,
"name": "Compton",
"cod": 200
}
{
"datetime": "2024-05-26 08:53:33",
"coord": { "lon": -1.3989, "lat": 51.0267 },
"base": "stations",
"main": {
"temp": 14.18,
"temp_min": 12.99,
"temp_max": 15.45,
"pressure": 1009,
"humidity": 93
},
"visibility": 10000,
"wind": { "speed": 4.12, "deg": 200 },
"clouds": { "all": 75 },
"sys": {
"type": 2,
"id": 2005068,
"country": "GB"
},
"timezone": 3600,
"id": 2652491,
"name": "Compton",
"cod": 200
}
{
"datetime": "2024-05-26 08:51:33",
"coord": { "lon": -1.3989, "lat": 51.0267 },
"base": "stations",
"main": {
"temp": 14.17,
"temp_min": 12.99,
"temp_max": 16,
"pressure": 1009,
"humidity": 90
},
"visibility": 10000,
"wind": { "speed": 4.63, "deg": 200 },
"clouds": { "all": 75 },
"sys": {
"type": 2,
"id": 2005068,
"country": "GB"
},
"timezone": 3600,
"id": 2652491,
"name": "Compton",
"cod": 200
}
...
Use Event Processing to...
Aggregate the weather measurements in hourly windows, generating statistical summaries of the measurements taken in this time.
{
"temperature min": 12.99,
"temperature max": 17.02,
"temperature average": 14.88,
"temperature standard deviation (population)": 0.451,
"pressure min": 1009,
"pressure max": 1009,
"pressure average": 1009,
"pressure standard deviation (population)": 0,
"humidity min": 84,
"humidity max": 90,
"humidity average": 87,
"humidity standard deviation (population)": 1,
"wind speed min": 4.12,
"wind speed max": 4.63,
"wind speed average": 4.39,
"wind speed standard deviation (population)": 0.254,
"cloudiness min": 75,
"cloudiness max": 75,
"cloudiness average": 75,
"cloudiness standard deviation (population)": 0,
"start": "2024-05-26 08:00:00",
"end": "2024-05-26 09:00:00"
}
Use watsonx to...
Summarize the numerical aggregate values in a simple English explanation.
During the one-hour period between 8:00 AM and 9:00 AM on May 26, 2024, the weather in Compton, GB was characterized by a range of temperatures. The temperature fluctuated between 13.0 degrees Celsius and 17.0 degrees Celsius, with an average temperature of 14.9 degrees. This suggests that the day started off cool, but warmed up throughout the hour.
The pressure remained relatively constant at 1009 millibars, with no significant changes throughout the hour. This consistent pressure indicates stable weather conditions, which would make it feel comfortable for outdoor activities.
Humidity levels varied between 84% and 90%, with an average humidity of 87%. High humidity can make the air feel sticky and uncomfortable, so these readings suggest that it might have been muggy outside during this time.
Wind speed ranged from 4.1 m/s to 4.6 m/s, with an average wind speed of 4.4 m/s. Moderate wind speeds like these can make it feel cooler, as they help to dissipate some heat from the body.
Lastly, the cloud cover remained constant at 75% throughout the hour. While the cloud cover didn't change, the sun did start to peek through around 9:00 AM, signaling the beginning of a brighter day.
Overall, the weather in Compton, GB on May 26, 2024, was marked by a mix of temperatures, humidity, and cloud cover, with moderate wind speeds making it feel slightly cooler.
Output:
A Kafka topic with an event every hour containing an easy-to-understand summary of the weather in the last hour.
Demo:
Here is an example of a flow like this in action:
youtu.be/q9LBZiauf0Y
Building these flows
For more detail on how these flows were created, and tips for creating your own Event Processing flows with watsonx, see my detailed walkthrough.
#Highlights-home