Apptio for All

 View Only

 Download only delta records from source system

Jump to  Best Answer
  • ApptioforAll
Pranav Pathak's profile image
Pranav Pathak posted Wed March 29, 2023 08:51 AM

Hi Community - Is there an option to perform delta refresh ? 

E.g. - A table has 10 records and in the next refresh we have 2 new records coming in. Is there an option to just append the 2 new records and use the  10 from the existing table? 

I can see an APPEND option in datalink in the transform option ( destination step). Will this give me the required result? 

Thanks in advance.


#ApptioforAll
Apptio Community Member's profile image
Apptio Community Member posted Thu March 30, 2023 01:25 AM  Best Answer

Hi Pranav,

the append option will certainly work however you may need to be careful with how you set the upload cadence (ie. monthly vs ad hoc). In this situation you may want to consider having 1 job at the start of the month with overwrite to load in all of the known records at that time, and use the append function for when the delta arrive as a subsequent job.

The other thing to consider is maybe creating a unique ID on your data and using the Remove Duplicates pipeline step in case the same data gets uploaded twice if there are source system issues.

Regards Mark


#ApptioforAll
Guillermo Cuadrado's profile image
Guillermo Cuadrado posted Thu March 30, 2023 02:47 AM

I have not tried it, @Pranav Pathak

The advice/tips provided by @Mark Johnson are excellent. Nothing for me to add.


#ApptioforAll

Related Content