Cloud Pak for Data Group

How to submit a Consistent Region application using Redis as the checkpoint store 

Mon September 28, 2020 06:25 PM

p>If you are planning to use Consistent Region Applications in Streams 4.0, you will need to setup the Checkpoint Repository Store. There are 2 options : file system or Redis. If your Streams installation does not use a shared file system you will need to Redis. Also, you might prefer to use Redis to take advantage of Replication and/or Sharding. If you are not familiar with Redis, check out this article first: How to install and configure Redis

First, make sure your redis server is running. In my case, redis is running on host d0428b06, port 6400:

 > hostname
d0428b06.pok.hpc-ng.ibm.com

>ps -ef|grep redis
ivan    5469     1  0 Feb09 ?        00:04:47 ./redis-server *:6400

Assuming you already have your Streams instance created. In my example, I will be using the default instance StreamsInstance:

> streamtool lsinstance
StreamsInstance

Set the checkpointRepository configuration parameter to redis:

> streamtool setproperty instance.checkpointRepository=redis
CDISC0009I The instance.checkpointRepository property was set to the following value: "redis". The previous property value was "notSpecified". This change affects the StreamsInstance instance in the domain_ivan domain when the instance is restarted.

Now set the checkpointRepositoryConfiguration to specify the host and port where Redis is running:

 > streamtool setproperty instance.checkpointRepositoryConfiguration="{\"replicas\" : 1, \"shards\" : 1, \"replicaGroups\" : [{ \"servers\" : [\"d0428b06:6400\"], \"description\" : \"rack1\" }]}"
CDISC0009I The instance.checkpointRepositoryConfiguration property was set to the following value: "{"replicas" : 1, "shards" : 1, "replicaGroups" : [{ "servers" : ["d0428b06:6400"], "description" : "rack1" }]}". The previous property value was "". This change affects the StreamsInstance instance in the domain_ivan domain when the instance is restarted.

Start or restart your instance and let’s check if the parameters were set correctly:

 > streamtool getproperty -a|grep checkpoint
instance.checkpointRepository=redis
instance.checkpointRepositoryConfiguration={"replicas" : 1, "shards" : 1, "replicaGroups" : [{ "servers" : ["d0428b06:6400"], "description" : "rack1" }]}

You can now try to run one of the consistent region sample applications.

>cd $STREAMS_INSTALL/samples/spl/feature/ConsistentRegion
>make
>streamtool submitjob output/MultipleSources/sample.MultipleSources.sab
CDISC0079I The following number of applications were submitted to the StreamsInstance instance: 1. The instance is in the domain_ivan domain.
CDISC0080I The 0 job was submitted for the following application: sample.MultipleSources.sab. The job was submitted to the StreamsInstance instance in the domain_ivan domain.
CDISC0020I Submitted job IDs: 0

Check if the job is healthy:

 > streamtool lsjobs
Instance: StreamsInstance
  Id State   Healthy User   Date                     Name                      Group
   0 Running yes     ivan 2015-02-17T11:21:43-0500 sample::MultipleSources_0 default

And check if output file is being generated:

> ls -al ./data
total 8
drwxr-xr-x 2 ivan ccgroup   40 Feb 17 11:21 .
drwxr-xr-x 6 ivan ccgroup 4096 Feb 17 11:18 ..
-rw-r--r-- 1 ivan ccgroup 3404 Feb 17 11:22 multipleSources.dat

That’s it. The job is healthy and the output file is being generated. You succeed running your first Consistent Region application using Redis as the checkpoint store.

Statistics

0 Favorited
3 Views
0 Files
0 Shares
0 Downloads