Co-author – Srihari Ananda Kumar 

Amazon Kinesis is an Amazon Web Service (AWS) used for processing big data in real-time. It is a managed, scalable, cloud-based service that enables the processing of large volumes of streaming data in real time.

Building a flow in IBM App Connect with Amazon Kinesis

Use App Connect to build flows that integrate with Amazon Kinesis and other applications. The connector is displayed as Amazon Kinesis on the App Connect User Interface (UI). 

To allow App Connect to connect to your Amazon Kinesis account, you need to complete the connection fields that you see in the App Connect Designer Catalog page or flow editor.

For more information on connection fields, see How to use App Connect with Amazon Kinesis .

Amazon Kinesis objects

The following are the Amazon Kinesis objects that can be run in App Connect.

Scenario 1: Filter records with speed limit violations and add these records to Amazon S3 via the Amazon Kinesis delivery stream

Consider this scenario: Amazon Kinesis filter records with speed limit violations and these records are added to Amazon S3 storage via the Amazon Kinesis delivery stream. The Amazon CloudWatch event logs are seamlessly retrieved and published to an Amazon Kinesis data stream. The data is processed, and the filtered data (which violates the speed limit) is added to the Amazon S3 bucket via Amazon Kinesis delivery streams.

Before you begin:

This template helps the user filter vehicle data for speed violations. To use this template, you need to ensure that the required data is configured in Amazon CloudWatch and the Kinesis platform.

1. Amazon CloudWatch collects raw data of vehicles and their information.
   In this example, a speed sensor application is used, which records the speed of the vehicle as it crosses the sensor. It logs the data from the sensor application, which records the speed and other information of the vehicle as it crosses the sensor, such as vehicle ID, owner ID, and speed, to Amazon CloudWatch.
   
   
2. Amazon Kinesis is configured with a data stream that processes these records, filters the vehicle records for speed limit violations, and adds these records to Amazon S3 storage via the Kinesis delivery stream.
   In this example, a data stream is configured to retrieve the Amazon CloudWatch event logs, which are retrieved continuously and pushed to a data stream in Amazon Kinesis.
   The Kinesis Analytics applications filter the data where the speed is above 80 km/h and store the data in Amazon S3 via delivery streams. The application must be kept running.

Note:

For more information on setting up data streams, refer to the following reference links, which will help you understand Amazon Kinesis Data Streams:

·         Amazon Kinesis Data Streams

·         What Is Amazon Kinesis Data Streams?

·         Creating and Managing Streams

·         Create a Kinesis data stream (Streaming Analytics Workshop)

In this flow:

1. You run a scheduler-based flow that starts the Amazon Kinesis application on an hourly basis. The application (that has been configured in Amazon Kinesis) must be kept running.
2. Amazon CloudWatch retrieves the filtered log events from the configured log group and stream.
   
   
   
    The Amazon CloudWatch event logs are retrieved and published to an Amazon Kinesis data stream.
   
   
3. Amazon Kinesis adds (with the Add multiple records operation) the vehicle-related records for those vehicles that have a speed violation of above 80 km/h and stores the data in Amazon S3 via delivery streams.
   
   
   
   To summarize: 
   The Amazon CloudWatch logs are retrieved continuously and pushed to the specified data stream in Amazon Kinesis. The Analytics applications then filter the data where the vehicle speed is above 80 km/h and add the records with the vehicle information to Amazon S3 via the specified delivery streams.
   
   
4. A Slack message gets sent to the intended channel to notify them that the vehicle record has been added and a summons can now be sent.
   
   Note: You can now use this filtered data, which has the details of the vehicle owners, to send summonses from an external application to the respective owners.

Resources

Try out our templates

You can view the following use cases in the Templates gallery in your App Connect Designer instance.

• Filter records with speed limit violations and add these records to Amazon S3 via the Amazon Kinesis delivery stream
  Template URL: https://<your-instance-id>/templates/ Filter%20records%20with%20speed%20limit%20violations%20and%20add%20these%20records%20to%20Amazon%20S3%20via%20the%20Amazon%20Kinesis%20delivery%20stream
• Data integration between Microsoft Azure Blob storage and Amazon Kinesis
  Template URL: https://<your-instance-id>/templates/Data%20integration%20between%20Microsoft%20Azure%20Blob%20storage%20and%20Amazon%20Kinesis
• Get Amazon Kinesis data stream records
  Template URL: https://<your-instance-id>/templates/Get%20Amazon%20Kinesis%20data%20stream%20records
• Add records to Amazon Kinesis data streams for each object retrieved from Amazon S3
  Template URL: https://<your-instance-id>/templates/Add%20records%20to%20Amazon%20Kinesis%20data%20streams%20for%20each%20object%20retrieved%20from%20Amazon%20S3

You must enable the Designer AI features in your containerized environment in order to access the App Connect templates. For more information, see The preloaded IBM App Connect templates.

You can also import these use cases directly into your App Connect Designer. These templates are added to a public GitHub repository at https://github.com/ot4i/app-connect-templates/tree/cp4i-templates/resources.

For more information about the supported connectors, see [App Connect Connectors].