Our TBM team has grown over the past year, and with a group of 5 of us, we are looking to develop a more defined process on Data table updates.

Just wondering what processes other companies are doing to provide control over data?

Lisa  

We have a smaller sub team (Data Team) focused on getting data into CT each month.  They work from a master data catalog (we're close to 100 source files now) that describes source/destination/owner/when available/when to load/etc.

We use a set of Box folders to shepherd the process.  Data owners are expected to place their files in an Intake folder in Box.  From there, the Data Team moves it through any pre load validation steps on the way to actually loading the files.  A series of Box folders exist to follow a file through those steps and tracking each file through that process.   (and of course chase any stragglers)

It's far from perfect and we are actively working a number of improvements (e.g. source file consolidation where feasible, more automation, fewer handoffs, additional validations) but it's a whole lot better than it was a year ago.  

I'm curious as well to see how others handle this.  

Thanks for your advice George!

Great suggestions Tony - thanks

Hi Lisa,

@Tony Wong has given a fantastic process for managing data and this is very similar to how Apptio in Australia is providing Managed Services to a number of smaller clients. These are organisations that typically do not have the resources to review and validate the data and outsource this to Apptio to manage.

You might in addition to Tony's suggestions set up a spreadsheet for all of your tables, which you complete on a monthly basis. This would include Table Name, Data Owner, Status, Due Date, Column count, Row Count, Cadence, Required, Data Source, General notes, etc. That way you can easily track the data as it comes in, if it has been validated, who needs to be chased, etc. Save copies of these so you can build up a history of your monthly practice so any new team members can easily see the process. I have attached an example for you - please excuse the formatting as it has been downloaded and exported from the online tool we use, but it should hopefully give you the general idea for setting one up for yourself.

Regards Mark

Thanks Mark!  Appreciate your advice.  

@Lars Peterson, I did start with the Data Expiration report in Data Quality, and am interested to utilize this as part of the process, and love the modifications you have made !!

Thank you both!!

Also using Data Expiration in Data Quality, and adding important comments and data provider name (similar to what Mark captures in a spreadsheet) in each table's Source transform pipeline step, as those notes show up as a column in Data Expiration.  This reduces the need to create a tracker outside of the system.

Hi Lisa,

We have, like @Mark Johnson, created a spread sheet in excel to see what status our uploads are in. It's based on the Data Expiration report in Data Quality. We made some modifications to fit the purpose and added some additional information, like if the upload is expected to be done manual or with Datalink or if it is just an appended data set and therefore don't need any action.

We just keep it simple to color the month when we have done our upload (Green/Yellow/Red) just to easy show the status. 

We have only used this since the beginning of the year, so it will most certainly evolve :-)

Regards Lars

We are using a similar excel tracking solution but I think it onerous and cumbersome to keep updated month on month. Is anyone considering eventually going back to the Data Quality reports (perhaps customizing them to add all the extra details we add to excel) so that this can be more automated? 

This was a great question and it's interesting to see how other folks are grappling with it.  You're seeing best practices emerge in the answers:  (e.g. catalog and attribute your data sources whether it be excel or elsewhere and establish ownership of those data sources).  

@Tony Wong had another important point about understanding your cycle/cadence and planning with that in mind.  We have a number of source files that are point in time views and don't have to be pulled at the end of the month.  We try to pull and validate them earlier in the month to lessen the madness as month end nears.  An example would be our Server to Application mappings.  We believe that as long as we are pulling it on a consistent date month over month, the risk of missing a few new mappings at the end of the month is outweighed by the benefits of having fewer moving parts at month end.  

We try and use Data Link wherever possible so we are not reliant on people providing the data for us. We have found in the past that when the data owners change the source data it causes us real headaches whilst in the middle of data load. Having the data link eliminates this. Where we cant use a data link we upload an excel file manually, we also use an excel spread sheet to track the progress of the monthly data load. We have started to introduce Data Supply Agreements (DSA), this agreement ensures that the data is supplied on time and in the correct format on a monthly basis, it also ensures that the owner of the data is aware that we use the data in Apptio and if any changes are to be made to the report we are given advance warning, instead of it being a surprise.