Cognos Analytics

Cognos Analytics

Connect, learn, and share with thousands of IBM Cognos Analytics users! 

 View Only

  • 1.  Incremental Upload to Dataset

    Posted Tue March 24, 2020 06:31 PM
    Hi All,

    As most of other tools provide feature for incremental update to Extract, Is it somehow possible to do with Dataset in new release?

    Thanks!

    ------------------------------
    Buddhi Gupta
    ------------------------------

    #CognosAnalyticswithWatson


  • 2.  RE: Incremental Upload to Dataset

    Posted Wed March 25, 2020 03:21 AM
    This is not possible for now but it will be a very good idea. The issue here is how to orginse it to know where the data is freezed and which data needs to be added as the filters and everything is defined in the dataset itself. Generally its a sql. If you want it you will need actully 2 datasets 1 which is static and adding the data into it and one where the refesh data is coming from. Its a good idea for CAOP but it will not be easy to be created.

    ------------------------------
    Stoyka Bojkova
    ------------------------------

    Original Message


  • 3.  RE: Incremental Upload to Dataset

    Posted Wed March 25, 2020 11:06 AM

    Incremental updates are possible, but they're fiddly to get working.

    Step 1. Make the history dataset. This is the first load that generates the base table. 
    Step 2. Make the current dataset. This is what will source the incremental update.
    Step 3. Make a data module, importing the two datasets previously made. Make sure to give them meaningful identifiers.
    Step 4. Make a union custom table between the two datasets. 
    Step 5. Make a dataset based on that union.
    Step 6. Import the new union dataset into the data module and give that a unique identifier. 

    This is where it gets tricky. The DM interface doesn't let you replace items in a union. 
    Step 7. Copy the unique identifier from the history dataset feeding the union. 
    Step 8. Remove the history dataset - this should cause a validation error.
    Step 9. Replace the identifier of the union dataset with the identifier you copied from the history set. This will replace the missing table in the union and cause the validation error to go away.  

    Now when you refresh the union dataset, it will union itself to the current dataset, effectively an incremental update.



    ------------------------------
    Paul Mendelson
    ------------------------------

    Original Message


  • 4.  RE: Incremental Upload to Dataset

    Posted Thu March 26, 2020 01:42 PM
    Thanks a lot @Paul Mendelson for solution!  

    Do we need to do something different for identify any changes in existing data and update for those records along with adding new data?​

    Example: to see if a table structure is changed and as well as any new table added to database. 

    Thanks for any inputs!

    ------------------------------
    Buddhi Gupta
    ------------------------------

    Original Message


  • 5.  RE: Incremental Upload to Dataset

    Posted Thu March 26, 2020 02:29 PM
    This really is a super fragile solution. Any changes to the structure would require a change to the datasets, and probably a rebuild. 

    There is a push to open and extend the APIs to the datasets (to insert, update, delete) but I'm not sure if they've been released publicly yet.

    ------------------------------
    Paul Mendelson
    ------------------------------

    Original Message


  • 6.  RE: Incremental Upload to Dataset

    Posted Mon May 22, 2023 11:36 AM

    Enhancement, add your votes and comments.

    Ability to append data to data set

    https://ibm-data-and-ai.ideas.ibm.com/ideas/CAOP-I-562



    ------------------------------
    DEAN BLACK
    ------------------------------

    Original Message


  • 7.  RE: Incremental Upload to Dataset

    Posted Tue May 23, 2023 08:58 AM

    Voted. Not sure if you noticed, but it does have a Status: Not under consideration

    This would be a really useful feature to have for sure. I believe both PowerBI and Tableau have had this in their products for years now. 



    ------------------------------
    Sanish John
    ------------------------------

    Original Message


  • 8.  RE: Incremental Upload to Dataset

    Posted Tue June 06, 2023 10:02 AM

    Definitely not something I have tested but you might be able to accomplish this with a Jupyter notebook that "updates" a dataset: https://www.ibm.com/docs/en/cognos-analytics/11.1.0?topic=notebooks-writing-data-data-source. Within the notebook you could even read in the existing dataset to determine missing records that need to be added if that were a requirement. Manipulating large data frames in Pandas is usually fairly speedy but might have to be tested.



    ------------------------------
    Robert Hofstetter
    ------------------------------

    Original Message