- Large data storage requirements
In this article we demonstrate in a step-by-step manner how to achieve good performance under heavy loads, such as ODM sending 1000 10KB events per second to BAI.
The pipeline is the following: each CP4A components sends its own events to Kafka topics. The events are then processed (and transformed if necessary) by Flink, which puts the results into an Elasticsearch instance (or into your own data lake like HDFS or into another Kafka using Kafka egress). Finally, you can visualize the Elasticsearch data in Business Performance Center (or Kibana).
This article will help you to address the potential bottlenecks in this pipeline.
For instance, if you see a lot of "offset lag" (the difference between the total number of messages in Kafka and the number of messages that remain to be processed), you can tweak Flink & Elasticsearch to avoid this lag. If you see that there are event delays in Kafka, then you should review your Kafka configuration.
Steps to Install and Configure your Environment
This will allow you to have 3 Kafka brokers that use secure connections.
It's best to install Event Streams in the same namespace as BAI. This will result in a simpler configuration.
Keep in mind that you can also bring your own Kafka if you want. The same requirements apply; you'll need multiple brokers and secure connections for a production-ready environment.
For each component, you can manually set the number of Flink processing jobs that are going to read the related Kafka topics. "Parallelism" is constrained by the number of partitions in the Kafka topics (parallelism is less than or equal to the number of Kafka Topic partitions).
It is configured in each component's CR file.
You will then see new pods named <cr_instance_name>-bai-flink-taskmanager-<number_of_the_pod>.
Having multiple Flink jobs running in parallel on the same Kafka topic will help speed up the ingestion of events from Kafka.
3) Set the number of Kafka partitions per topic
We recommend to manually create the Kafka topics before installing BAI. This way, you'll have full control of the topic properties instead of relying on BAI pre-defined settings.
The names of the topics are the same as those in your CR file (probably the same names as in your current demo environment), here is where you can set them.
- Use at least 3 partitions per topic (one per broker) x parallelism value. In our case, we'll pick 12 partitions (3*4 flink jobs).
- For the data retention, it's important to establish disk capacity requirements, while minimizing the risk of exceeding it.
- 3 Replicas, so that if a Kafka broker is down, the other 2 are capable of handling the load.
- Each topic should have the same settings as the others.
By using these settings, you'll improve performance for Kafka clients, which in our case are the CP4A components.
In some circumstances, you want to re-use your own Elasticsearch infrastructure, which is tailored to your needs and over which you have full control.
During our tests, we found that an external Elasticsearch performs better. The improved performance is due to attached disks being faster when compared to OCP cluster storage, which relies on NFS.
In our case, we picked a cluster with 3 master/data/client nodes, with a 2 HA Proxy load balancer in front of it, accessible via a Virtual IP.
This allows us to have HA for the Elasticsearch cluster. Each data node has a large disk capacity to store our events for a long period of time.
Configure your CR file in the following manner to enable BAI to use an external Elasticsearch:
You can either configure the full list of Kafka brokers or you can configure a single bootstrap url. This will allow the event emitters to take advantage of Kafka horizontal scaling without issues.
If something went wrong, it's probably because the json being sent is malformed.
From now on, for the new indexes created by BAI, you should have the correct number of shards. Note that due to an Elasticsearch limitation, you cannot edit the number of shards for an already existing index. So, either delete the already created indexes if you don't care about the data that they contain, or re-index all the indexes.
You can set fine permissions on the data presented in BPC. By creating a team of administrator users, you can then use an admin account to set the permissions for other teams.
a) Create a team in UMS following the Knowledge center procedure: https://www.ibm.com/support/knowledgecenter/en/SSYHZ8_20.0.x/com.ibm.dba.offerings/topics/con_ums_teams_manage.html
b) Modify your CR to set the UMS team as BPC admin, and make sure to not allow everyone
We hope that this guide will help you achieve your BAI performance and scalability goals in your production environment.