Topic Thread

Expand all | Collapse all

Incremental Upload to Dataset

  • 1.  Incremental Upload to Dataset

    Posted 8 days ago
    Hi All,

    As most of other tools provide feature for incremental update to Extract, Is it somehow possible to do with Dataset in new release?

    Thanks!

    ------------------------------
    Buddhi Gupta
    ------------------------------


  • 2.  RE: Incremental Upload to Dataset

    Posted 8 days ago
    This is not possible for now but it will be a very good idea. The issue here is how to orginse it to know where the data is freezed and which data needs to be added as the filters and everything is defined in the dataset itself. Generally its a sql. If you want it you will need actully 2 datasets 1 which is static and adding the data into it and one where the refesh data is coming from. Its a good idea for CAOP but it will not be easy to be created.

    ------------------------------
    Stoyka Bojkova
    ------------------------------



  • 3.  RE: Incremental Upload to Dataset

    Posted 8 days ago

    Incremental updates are possible, but they're fiddly to get working.

    Step 1. Make the history dataset. This is the first load that generates the base table. 
    Step 2. Make the current dataset. This is what will source the incremental update.
    Step 3. Make a data module, importing the two datasets previously made. Make sure to give them meaningful identifiers.
    Step 4. Make a union custom table between the two datasets. 
    Step 5. Make a dataset based on that union.
    Step 6. Import the new union dataset into the data module and give that a unique identifier. 

    This is where it gets tricky. The DM interface doesn't let you replace items in a union. 
    Step 7. Copy the unique identifier from the history dataset feeding the union. 
    Step 8. Remove the history dataset - this should cause a validation error.
    Step 9. Replace the identifier of the union dataset with the identifier you copied from the history set. This will replace the missing table in the union and cause the validation error to go away.  

    Now when you refresh the union dataset, it will union itself to the current dataset, effectively an incremental update.



    ------------------------------
    Paul Mendelson
    ------------------------------



  • 4.  RE: Incremental Upload to Dataset

    Posted 7 days ago
    Thanks a lot @Paul Mendelson for solution!  

    Do we need to do something different for identify any changes in existing data and update for those records along with adding new data?​

    Example: to see if a table structure is changed and as well as any new table added to database. 

    Thanks for any inputs!

    ------------------------------
    Buddhi Gupta
    ------------------------------



  • 5.  RE: Incremental Upload to Dataset

    Posted 7 days ago
    This really is a super fragile solution. Any changes to the structure would require a change to the datasets, and probably a rebuild. 

    There is a push to open and extend the APIs to the datasets (to insert, update, delete) but I'm not sure if they've been released publicly yet.

    ------------------------------
    Paul Mendelson
    ------------------------------