Platform

Platform

A place for Apptio product users to learn, connect, share and grow together.

 View Only

Monitoring Data Freshness of Table Uploads via Editable Table for dedicated table scope

By Karsten Leurs posted Wed January 24, 2024 04:13 AM

  

CT has internal tools for checking data freshness, e.g. the OOTB reports in the Data Quality section and also some tools within TBM Studio. However with some clients we found the need for a custom solution where we only trace a certain scope of tables with custom rules, so we are not confusing with full list of tables + be able to focus on our own freshness rules. 

In this article I will explain how to use editable table to dynamically monitor the upload status with traffic lights based on custom rules.

We will be able to

  • decide which tables are checked in this report (which also means, we have to enter them, so new tables have to be added)
  • decide the rules for flagging red, green (and if needed, yellow)
  • put this into report surface, potentially visible to all users

We define following validity rules:

Upload Date Criteria Explanation
Upload Date during this year The file has to be uploaded during the year (so it should not be older then the report year)
Upload Date after this month The file has to be uploaded after end of the report month (e.g. to ensure it has ultimo data)
Upload Date during this month The file can already be uploaded during the report month
Upload into this month Monthly versioned table, we just expect a file in report month (no check when it was uploaded)
No Check no check will be performed - no traffic light

TBM Studio of the editable table report

Steps:

  1. Create "Editable Table" with e.g. four label columns:
    • Dataset Name
    • Category
    • Upload Date Criteria
      In here we define Possible Values:
      Upload Date during this year,Upload Date after this month,Upload Date during this month,Upload into this month,No Check
    • Comment
      Editable Table Configure Columns


  2. Create the new report and insert the Editable Table component with those four columns
  3. Create the following formula columns into the report table via Data -> Insert Formula Column:

    Formula column Name Formula
    Upload Date =DateFormat(TableInfo("Last Updated",{Dataset Name}), "yyyy-MM-dd")
    File Name =TableInfo("Source",{Dataset Name})
    Month Current =Months(CurrentDate())
    Month Upload =Split(Months(Upload Date),1,".")
    Month Mininum for Green for Upload Date =If({Upload Date Criteria}="Upload Date after this month",Month Current+1,If({Upload Date Criteria}="Upload Date during this month",Month Current,If({Upload Date Criteria}="Upload Date during this year",Month Current-CurrentDate("M")+1,0)))
    Uses Data from This Period =TableInfo("Updated This Period",Dataset Name)
    Traffic Light Color =If({Upload Date Criteria}="No Check" OR {Upload Date Criteria}="","No Check",If((Upload Date Criteria="Upload into this month" AND Uses Data from This Period="true") OR (Upload Date Criteria!="Upload into this month" AND Month Upload>=Month Mininum for Green for Upload Date),"green","red"))
    Upload Status ="<div align='center'>"&Icon("3colorcircles",Traffic Light Color="green",Traffic Light Color="yellow",Traffic Light Color="red")
  4. Fill the table with some table names, categories, and upload rules. Validate its working.
  5. Hide the columns marked in yellow in following printscreen in the report table.
Notes:
  • There is no need to publish this editable table into a CT table. It works just off the editable table itself. So the Publish Feature should be disabled for all roles in the editable-report component.
  • If this report is also visible to Non-Admins, make sure to set appropriate permissions to the editable table report component (e.g. Add/Change Row etc).

#design-patterns #design #reporting #DataFreshness #Upload #Monitoring


#TBMStudio
0 comments
16 views

Permalink