Cloud Pak for Data Group

List of data sources supported by Streams 

Thu October 01, 2020 09:05 PM

A very common question from Streams users is whether Streams can connect to a particular data source to read or write data. In this article,  I’ll summarize the Streams toolkits that provide access to some of the more common data sources.

What is a toolkit?
 A toolkit is a package of artifacts that provide some function. For example, to connect to Apache Kafka, you would use the Streams Kafka toolkit. Streams provides dozens of toolkits for analyzing and ingesting data.

The following lists are evolving, as additional toolkits might be added to the Streams open source project on GitHub.

The first list below is of the toolkits that are included with Streams on Cloud Pak for connecting to data sources. 
The second is of toolkits on GitHub that are available in various stages of maturity. To use these toolkits you would have to download them from GitHub.

See below for next steps if your data source is not listed here.

Where listed, Python packages are on  PyPi and installable with pip.

Data sources supported by default in Streams 5.3+

Data Source Toolkit

Apache Kafka

Kafka Toolkit:

https://github.com/IBMStreams/streamsx.kafka
Python Package: streamsx.kafka

IBM Event Streams

­­Message Hub toolkit

https://github.com/IBMStreams/streamsx.messagehub

Python package: streamsx.kafka

JDBC databases : IBM Db2, IBM Db2,
ORACLE, Teradata, PostgresSql, Microsoft SQL Server, MySql

JDBC Toolkit

https://github.com/IBMStreams/streamsx.jdbc

Python package: streamsx.database

Db2 Event Store

Event Store toolkit­­­https://github.com/IBMStreams/streamsx.eventstore

Python package: streamsx.eventstore

Apache Hbase

HBase toolkit

https://github.com/IBMStreams/streamsx.hbase

Python package: streamsx.hbase

RabbitMQ

Rabbit MQ Toolkit

https://github.com/IBMStreams/streamsx.rabbitmq

MQTT

https://github.com/IBMStreams/streamsx.mqtt

Python package: streamsx.mqtt

IBM MQ, ActiveMQ

JMS toolkit

https://github.com/IBMStreams/streamsx.jms

Python package: streamsx.jms

Network data

https://github.com/IBMStreams/streamsx.network

HTTP endpoints/REST/Websockets

Inet toolkit

https://github.com/IBMStreams/streamsx.inet

Python package: streamsx.inet

XMS

Messaging toolkit

https://github.com/IBMStreams/streamsx.messaging

Elasticsearch

https://github.com/IBMStreams/streamsx.elasticsearch

Python package: streamsx.elasticsearch

HDFS

https://github.com/IBMStreams/streamsx.hdfs

Python package: streamsx.hdfs

Object Storage

https://github.com/IBMStreams/streamsx.objectstorage

Python package: streamsx.objectstorage

Additional toolkits available on GitHub


Data source

Toolkit

Cassandra

https://github.com/IBMStreams/streamsx.cassandra

Cloudant

https://github.com/IBMStreams/streamsx.cloudant

Watson Internet of Things Platform

https://github.com/IBMStreams/streamsx.iot

MongoDB

https://github.com/IBMStreams/streamsx.mongoDB

Websockets

https://github.com/IBMStreams/streamsx.websocket

Slack

https://github.com/IBMStreams/streamsx.slack

Watson Explorer

https://github.com/IBMStreams/streamsx.watsonexplorer

InfoSphere Data Replication Change Data Capture

https://github.com/IBMStreams/streamsx.cdc

Streaming Analytics service in IBM Cloud

The Streaming Analytics service in the public IBM Cloud supports a subset of the aforementioned toolkits. View the full list of supported toolkits in the Streaming Analytics service.


Don't see your database or system?

  • If you do not see the data source you are looking for in this list, please search the Streams GitHub home page. If there is no project, you can either:

Useful links

Resources for Streams developers

Statistics

0 Favorited
21 Views
0 Files
0 Shares
0 Downloads