Cloud Pak for Data

 View Only

  • 1.  ICP for Data as Platform for Data Lake?

    Posted Fri July 13, 2018 03:48 AM
    Hi.

    I am a solution builder.
    I am looking for a platform with the following capabilities:
    1. powerful data catalog capability because the solution requires to ingest time series data at 10 billion records per hour
    2. data store for this data lake that is growing by 10 billion records per hour
    3. integrate with applications I will deliver as content on ICP (not Data)
    4. streaming ETL for real time analytics
    5. historical ETL for machine learning training
    How many of the above requirements can be fulfilled by ICP for Data?
    Thanks
    YH

    ------------------------------
    YH Lim
    ------------------------------

    #CloudPakforDataGroup


  • 2.  RE: ICP for Data as Platform for Data Lake?

    Posted Fri July 27, 2018 01:54 PM
    Hi YH,

      Please see my responses below.

    Q) powerful data catalog capability because the solution requires to ingest time series data at 10 billion records per hour 
    A) We plan to have this capability in Q318 as an add on through Db2 Event Store. 

    Q) data store for this data lake that is growing by 10 billion records per hour
    A) With the large growth rate, Db2 Event Store is a fine tuned data store that can serve the queries to navigate data.  This architecture would also allow to separate compute from storage.  Db2 Event Store is well suited for this and will be available as an add on per previous update.

    Q) integrate with applications I will deliver as content on ICP (not Data)
    A) Yes, all ICP for Data databases are enabled for external access so they can be integrated at any level

    Q) streaming ETL for real time analytics
    A) At this time partly supported as part of the transform capability.  Streaming connectors are still being reviewed.  Can you please help provide use cases that you may have for our benefit of understanding? 

    Q) historical ETL for machine learning training
    A) Need more info here. Can you please help provide use cases that you may have for our benefit of understanding?

    Q) How many of the above requirements can be fulfilled by ICP for Data? 
    A) See above

    ------------------------------
    Roger Hong
    ------------------------------

    Original Message