How to submit a Consistent Region application using Redis as the checkpoint store

If you are planning to use Consistent Region Applications in Streams 4.0, you will need to setup the Checkpoint Repository Store. There are 2 options : file system or Redis. If your Streams installation does not use a shared file system you will need to Redis. Also, you might prefer to use Redis to take advantage of Replication and/or Sharding. If you are not familiar with Redis, check out this article first: How to install and configure Redis

First, make sure your redis server is running. In my case, redis is running on host d0428b06, port 6400:

 > hostname
d0428b06.pok.hpc-ng.ibm.com

>ps -ef|grep redis
ivan    5469     1  0 Feb09 ?        00:04:47 ./redis-server *:6400

Assuming you already have your Streams instance created. In my example, I will be using the default instance StreamsInstance:

> streamtool lsinstance
StreamsInstance

Set the checkpointRepository configuration parameter to redis:

> streamtool setproperty instance.checkpointRepository=redis
CDISC0009I The instance.checkpointRepository property was set to the following value: "redis". The previous property value was "notSpecified". This change affects the StreamsInstance instance in the domain_ivan domain when the instance is restarted.

Now set the checkpointRepositoryConfiguration to specify the host and port where Redis is running:

 > streamtool setproperty instance.checkpointRepositoryConfiguration="{\"replicas\" : 1, \"shards\" : 1, \"replicaGroups\" : [{ \"servers\" : [\"d0428b06:6400\"], \"description\" : \"rack1\" }]}"
CDISC0009I The instance.checkpointRepositoryConfiguration property was set to the following value: "{"replicas" : 1, "shards" : 1, "replicaGroups" : [{ "servers" : ["d0428b06:6400"], "description" : "rack1" }]}". The previous property value was "". This change affects the StreamsInstance instance in the domain_ivan domain when the instance is restarted.

Start or restart your instance and let’s check if the parameters were set correctly:

 > streamtool getproperty -a|grep checkpoint
instance.checkpointRepository=redis
instance.checkpointRepositoryConfiguration={"replicas" : 1, "shards" : 1, "replicaGroups" : [{ "servers" : ["d0428b06:6400"], "description" : "rack1" }]}

You can now try to run one of the consistent region sample applications.

>cd $STREAMS_INSTALL/samples/spl/feature/ConsistentRegion
>make
>streamtool submitjob output/MultipleSources/sample.MultipleSources.sab
CDISC0079I The following number of applications were submitted to the StreamsInstance instance: 1. The instance is in the domain_ivan domain.
CDISC0080I The 0 job was submitted for the following application: sample.MultipleSources.sab. The job was submitted to the StreamsInstance instance in the domain_ivan domain.
CDISC0020I Submitted job IDs: 0

Check if the job is healthy:

 > streamtool lsjobs
Instance: StreamsInstance
  Id State   Healthy User   Date                     Name                      Group
   0 Running yes     ivan 2015-02-17T11:21:43-0500 sample::MultipleSources_0 default

And check if output file is being generated:

> ls -al ./data
total 8
drwxr-xr-x 2 ivan ccgroup   40 Feb 17 11:21 .
drwxr-xr-x 6 ivan ccgroup 4096 Feb 17 11:18 ..
-rw-r--r-- 1 ivan ccgroup 3404 Feb 17 11:22 multipleSources.dat

That’s it. The job is healthy and the output file is being generated. You succeed running your first Consistent Region application using Redis as the checkpoint store.