Platform

Hi, 

Can we automate the process of eliminating few columns that flow into tables in CT ?

Eg: We have 200+ columns related to cloud that are flowing into Azure raw table, which we don't need anymore. Is there any way we can create a new column with a formula that eliminates those columns, so that we then use that column in the filter step to eliminate these columns?

We can use Hide and rename to hide them but, since every month some new columns are getting added, how to go about it?

Are you using the Cloudability Datalink connection or the MCC/Datalink connection?  If MCC/Datalink, we have the ability to include/exclude columns in the configuration which determines what will be ingested into TBM Studio. 

More broadly, we are investigating a pre-processor that will filter out columns, rows (perform groupings and other functions) before a dataset is ingested into TBM Studio.  The pre-processor configuration would be part of the the data pipeline.  

Original Message:
Sent: 10-18-2023 12:13
From: Nitish Collooru
Subject: Can we automate the process of eliminating few columns that flow into tables in CT ?

Hi, 

Can we automate the process of eliminating few columns that flow into tables in CT ?

Eg: We have 200+ columns related to cloud that are flowing into Azure raw table, which we don't need anymore. Is there any way we can create a new column with a formula that eliminates those columns, so that we then use that column in the filter step to eliminate these columns?

We can use Hide and rename to hide them but, since every month some new columns are getting added, how to go about it?

Hi, 

Can we automate the process of eliminating few columns that flow into tables in CT ?

Eg: We have 200+ columns related to cloud that are flowing into Azure raw table, which we don't need anymore. Is there any way we can create a new column with a formula that eliminates those columns, so that we then use that column in the filter step to eliminate these columns?

We can use Hide and rename to hide them but, since every month some new columns are getting added, how to go about it?

It doesn't really eliminate them to your specific request, but have you thought about doing an append from this table into another and work from there? that way you can choose the fields you append and can ignore the rest.

Hi, 

Can we automate the process of eliminating few columns that flow into tables in CT ?

Eg: We have 200+ columns related to cloud that are flowing into Azure raw table, which we don't need anymore. Is there any way we can create a new column with a formula that eliminates those columns, so that we then use that column in the filter step to eliminate these columns?

We can use Hide and rename to hide them but, since every month some new columns are getting added, how to go about it?

It doesn't really eliminate them to your specific request, but have you thought about doing an append from this table into another and work from there? that way you can choose the fields you append and can ignore the rest.

Hi, 

Can we automate the process of eliminating few columns that flow into tables in CT ?

Eg: We have 200+ columns related to cloud that are flowing into Azure raw table, which we don't need anymore. Is there any way we can create a new column with a formula that eliminates those columns, so that we then use that column in the filter step to eliminate these columns?

We can use Hide and rename to hide them but, since every month some new columns are getting added, how to go about it?

Hi, 

Can we automate the process of eliminating few columns that flow into tables in CT ?

Eg: We have 200+ columns related to cloud that are flowing into Azure raw table, which we don't need anymore. Is there any way we can create a new column with a formula that eliminates those columns, so that we then use that column in the filter step to eliminate these columns?

We can use Hide and rename to hide them but, since every month some new columns are getting added, how to go about it?

It is also common to use a Hide / Rename step to exclude the unwanted columns. 

Hi, 

Can we automate the process of eliminating few columns that flow into tables in CT ?

Eg: We have 200+ columns related to cloud that are flowing into Azure raw table, which we don't need anymore. Is there any way we can create a new column with a formula that eliminates those columns, so that we then use that column in the filter step to eliminate these columns?

We can use Hide and rename to hide them but, since every month some new columns are getting added, how to go about it?

True, @Dan Kelly. As pointed out by @Nitish Collooru, new columns with tags keep popping into the file, so that approach is not sustainable.

It is also common to use a Hide / Rename step to exclude the unwanted columns. 

Hi, 

Can we automate the process of eliminating few columns that flow into tables in CT ?

Eg: We have 200+ columns related to cloud that are flowing into Azure raw table, which we don't need anymore. Is there any way we can create a new column with a formula that eliminates those columns, so that we then use that column in the filter step to eliminate these columns?

We can use Hide and rename to hide them but, since every month some new columns are getting added, how to go about it?