List of data sources supported by Streams

A very common question from Streams users is whether Streams can connect to a particular data source to read or write data. In this article,  I’ll summarize the Streams toolkits that provide access to some of the more common data sources.

What is a toolkit?
 A toolkit is a package of artifacts that provide some function. For example, to connect to Apache Kafka, you would use the Streams Kafka toolkit. Streams provides dozens of toolkits for analyzing and ingesting data.

The following lists are evolving, as additional toolkits might be added to the Streams open source project on GitHub.

The first list below is of the toolkits that are included with Streams on Cloud Pak for connecting to data sources. 
The second is of toolkits on GitHub that are available in various stages of maturity. To use these toolkits you would have to download them from GitHub.

See below for next steps if your data source is not listed here.

Where listed, Python packages are on  PyPi and installable with pip.

Data sources supported by default in Streams 5.3+

Data Source	Toolkit
Apache Kafka	Kafka Toolkit: https://github.com/IBMStreams/streamsx.kafka Python Package: streamsx.kafka
IBM Event Streams	­­Message Hub toolkit https://github.com/IBMStreams/streamsx.messagehub Python package: streamsx.kafka
JDBC databases : IBM Db2, IBM Db2, ORACLE, Teradata, PostgresSql, Microsoft SQL Server, MySql	JDBC Toolkit https://github.com/IBMStreams/streamsx.jdbc Python package: streamsx.database
Db2 Event Store	Event Store toolkit­­­https://github.com/IBMStreams/streamsx.eventstore Python package: streamsx.eventstore
Apache Hbase	HBase toolkit https://github.com/IBMStreams/streamsx.hbase Python package: streamsx.hbase
RabbitMQ	Rabbit MQ Toolkit https://github.com/IBMStreams/streamsx.rabbitmq
MQTT	https://github.com/IBMStreams/streamsx.mqtt Python package: streamsx.mqtt
IBM MQ, ActiveMQ	JMS toolkit https://github.com/IBMStreams/streamsx.jms Python package: streamsx.jms
Network data	https://github.com/IBMStreams/streamsx.network
HTTP endpoints/REST/Websockets	Inet toolkit https://github.com/IBMStreams/streamsx.inet Python package: streamsx.inet
XMS	Messaging toolkit https://github.com/IBMStreams/streamsx.messaging
Elasticsearch	https://github.com/IBMStreams/streamsx.elasticsearch Python package: streamsx.elasticsearch
HDFS	https://github.com/IBMStreams/streamsx.hdfs Python package: streamsx.hdfs
Object Storage	https://github.com/IBMStreams/streamsx.objectstorage Python package: streamsx.objectstorage

Additional toolkits available on GitHub

Data source	Toolkit
Cassandra	https://github.com/IBMStreams/streamsx.cassandra
Cloudant	https://github.com/IBMStreams/streamsx.cloudant
Watson Internet of Things Platform	https://github.com/IBMStreams/streamsx.iot
MongoDB	https://github.com/IBMStreams/streamsx.mongoDB
Websockets	https://github.com/IBMStreams/streamsx.websocket
Slack	https://github.com/IBMStreams/streamsx.slack
Watson Explorer	https://github.com/IBMStreams/streamsx.watsonexplorer
InfoSphere Data Replication Change Data Capture	https://github.com/IBMStreams/streamsx.cdc

Streaming Analytics service in IBM Cloud

The Streaming Analytics service in the public IBM Cloud supports a subset of the aforementioned toolkits. View the full list of supported toolkits in the Streaming Analytics service.

Don't see your database or system?

• If you do not see the data source you are looking for in this list, please search the Streams GitHub home page. If there is no project, you can either:
  • Submit an issue on the IBM Streams GitHub repository,
  • Or, write your own adapter to read from that system using the Python, C++ or Java API. This can be done if the data source has a client for those languages. You would create an operator that will read data using the client.

Useful links

Resources for Streams developers

#CloudPakforDataGroup