IBM Apptio

Apptio Community Member posted Wed February 03, 2021 04:54 PM

Hi Guillermo,

A couple clarifications would help me answer this one for you. Are you looking to pull BLOB data into Apptio? Or are you looking to analyze your existing storage servers with BLOB data present on them?

If you are looking to pull BLOB data into Apptio, it depends on what data it is.

Thanks,
Joe
#CostingStandard(CT-Foundation)

Guillermo Cuadrado posted Thu February 04, 2021 01:43 AM

The programs write CSV files and store them in a directory/folder in Azure BLOB storage.
Yes, I would like to get to those files (with Datalink or whatever) and load them into Cost Transparency.
They complement the billing information that we are already pulling in with the Muticloud connector in Datalink Classic.
#CostingStandard(CT-Foundation)

Guillermo Cuadrado posted Wed February 17, 2021 04:54 AM

OK, somebody gave me a REST API that returns something like this:

​​​
#CostingStandard(CT-Foundation)

Guillermo Cuadrado posted Wed September 29, 2021 07:53 AM

After much research and with the help of Apptio Support, we finally found out that Azure does not support REST access to files in a blob :-(
Does anybody have another idea?
#CostingStandard(CT-Foundation)

Jenny Franklin posted Wed September 29, 2021 09:36 AM

Checking to see if @Matt Temple knows 😁🐱‍👤🤩
#CostingStandard(CT-Foundation)

Apptio Community Member posted Wed September 29, 2021 03:48 PM

I created an ugly work-around for something similar. It works but required a bit of tuning over a couple months.

First, I created a Power Automate Flow which allowed me to connect to my storage BLOB (we have Azure AD - not sure if you can make this workk without that). The Flow copies the files in a directory to my OneDrive. On our Datalink server, I set up the OneDrive folder to sync. 

I also have a Powershell script scheduled to run every few hours on the Datalink server. It looks for the files in the OneDrive folder, and copies them to the location for Datalink to process.

There's probably a better way of handling this but it does (usually) work.
#CostingStandard(CT-Foundation)

Guillermo Cuadrado posted Thu September 30, 2021 02:05 AM

Thanks, @Robert Krauss. Sounds like a plan. That's what I'm doing—without the automation: I manually download the files with Azure Storage Explorer and then I group them in sets of 10 days. I have to do that since​ they are so big, that they exceed the maximum size of an Excel spreadsheet. Then I loaded the merged files, again by hand.
I'll start with configuring the local Datalink agent to send the files, and then I'll worry about the BLOB file access.
Sounds like a job for my alter ego, ​

Update 05Nov2021:

1) Still working on getting the files via CURL, or Python programs.
2) I wrote a program in Python that gets the files from the download area and compacts them as an output file.
3) Created a Datalink File System connector that picks up the files, uploads them to CT and then archives them afterwards.
Not 100% automated, but it's A LOT less labor-intensive.

#CostingStandard(CT-Foundation)

Prem Kilaru posted Mon August 26, 2024 11:45 AM

@Guillermo Cuadrado Did you try loading from Azure Files (Managed file shares for Azure cloud) - and then use File System connector? 

#CostingStandard(CT-Foundation)

Guillermo Cuadrado posted Tue September 03, 2024 10:50 AM

I have used the File System connector, but not for this purpose. In the meantime, our FinOps people have moved the data into a data lake. I still haven't managed to import the data automatically, as they are reluctant to provide a REST API. I have to go the manual way and download the data from a Power BI report by hand. Then Datalink kicks in. 

Thanks for the tip, @Prem Kilaru.

#CostingStandard(CT-Foundation)