IBM Apptio

IBM Apptio

 View Only

Ingest AWS Data (CT) 

Wed December 06, 2017 12:15 PM

♦  Applies to: Cost Transparency on TBM Studio 12.0 to 12.5.x

 

For CBM and Cloud on Cost Transparency 12.6 and later, see Apptio for Cloud Knowledge Base .

 

Use the following instructions to ingest data from your AWS environment. Apptio Cloud on Cost Transparency (CCM) relies on the following data sets.

 

NOTE:  For information about deprecated billing formats, see AWS Deprecation of Billing Formats.

 

AWS data setPurpose

Detailed Billing Report with resources and tags

Daily/Hourly Cost Transparency

Cost Allocation Report

Monthly Cloud TCO

EC2 Reserved Instances API

(DescribeReservedInstances)

EC2 RI Purchase Details

RDS Reserved Instances API

(DescribeDBReservedInstances)

RDS RI Purchase Details

Trusted Advisor APIs

(DescribeTrustedAdvisorChecks)

Cost Savings Recommendations

 

Steps

  1. Go to your Cloud Cost Management Datalink instance.
  2. Select the Default Agent.

Setup the current month daily billing ingestion (based on the Detailed Billing Report)

  1. Create a new AWS connector, and name it "DBR – Current Month."
  2. Select your preferred authentication method (Apptio recommends using Delegation for optimal security). 
    This step assumes you have already configured AWS to allow Datalink to authenticate and read data from your AWS environment. If not, see Complete AWS configuration dependencies.
  3. Enter your AWS authentication credential details.
  4. In Select data source, select Billing Report.
  5. In Report Settings, enter the name of the S3 bucket in which your AWS bills are placed.
  6. Leave the File Prefix blank.
  7. Select the Account Id for your AWS payer account.
  8. Select Detailed Billing Report with Resources and Tags (automatic transform).
  9. Skip the Column Mapping section for now. This will be revisited when mapping your tags/accounts to the Apptio schema.
  10. In AWS Time Period, select Current Month.
  11. In Apptio Destination, select R12 with Front Door for Apptio Version.
  12. In Apptio Instance, select Cost Transparency.
  13. In Domain, select your Apptio domain. This should be <your company name>.com.
  14. In Project, select Cost Transparency.
  15. Enter "AWS Billing" into Category.
  16. Enter "AWS" into Source System.
  17. Select Current Month in Time Period.
  18. In the Schedule section and select Enable Schedule.
  19. From the Frequency dropdown, select Daily.
  20. Select a Start Time that is early morning.
    This is a value that might need to be updated as you start to understand when your DBR values are updated.
  21. Test the connector.
  22. Save the connector.

Setup the prior month daily billing ingestion (this is to ensure you are getting the final bill for the previous month uploaded into the system)

  1. From the Connector list, select Copy from the Action menu for the DBR – Current Month connector.
  2. Change the name to "DBR – Prior Month."
  3. Change the two Time Period fields to Previous Month.
  4. Change the schedule to run a bit later in the day.
  5. Test the connector.
  6. Save the connector.

Setup the monthly billing ingestion (based on the Cost Allocation Report)

  1. From the Connector list, select Copy from the Action menu for the DBR – Prior Month connector.
  2. Change the name to "Cost Allocation Report."
  3. In Report Settings, change the Report Type to Cost Allocation Report.
  4. In Schedule, change the Frequency to Monthly value to a day close to the 5th of the month.
    This value might need to be changed as you figure out when AWS finalizes your bill.
  5. Test the connector.
  6. Save the connector.

Setup the EC2 reserved instance purchase connector

  1. From the Connector List, select Copy from the Action menu for the Cost Allocation Report connector
  2. Change the name to Reserved Instances – EC2
  3. In Select data source, change the selection to Reserved Instance Purchase Report
  4. In Reserved Instance Settings, select EC2
  5. Test the connector
  6. Save the connector

Setup the RDS reserved instance purchase connector

  1. From the Connector list, select Copy from the Action menu for the Reserved Instances – EC2 connector.
  2. Change the name to "Reserved Instances – RDS."
  3. In Reserved Instance Settings, select RDS.
  4. Test the connector.
  5. Save the connector.

Setup the Trusted Advisor connector

  1. From the Connector list, select Copy from the Action menu for the Reserved Instances – RDS connector.
  2. Change the name to "Trusted Advisor."
  3. In Trusted Advisor Settings, select Cost Optimizing and leave the other categories unchecked.
  4. Test the connector.
  5. Save the connector.

Run all the connectors

  1. Run each connector, leaving time for each to finish before running the next one.

 

(Back to top)

This article is open for your feedback. At the bottom of this page, click Add a comment.


#public cloud
#aws



#AWS
#CostingStandard(CT-Foundation)
#Costing
#CloudServices

Statistics
0 Favorited
1 Views
0 Files
0 Shares
0 Downloads

Comments

Mon April 29, 2019 08:02 PM

Hi Matt, 

Yes, the date is only announced for The Detailed Billing Report (DBR) and Detailed Billing Report with Resources & Tags (DBR-RT). 

https://docs.aws.amazon.com/awsaccountbilling/latest/aboutv2/detailed-billing-report-changes.html

I couldn't find any date for MCAR yet.....Cost Allocation Report will be unavailable at a later date:

https://docs.aws.amazon.com/awsaccountbilling/latest/aboutv2/reportwithtags.html

 

Regards,

Minoo.


#CostingStandard(CT-Foundation)

Mon April 29, 2019 04:23 PM

You're mentioning the DBR, but what about the MCAR (Monthly Cost Allocation Report)?

 

Many of the standard reports (up until recently) work best with the MCAR.

 

Clarity regarding the MCAR would be very appreciated. :-)


#CostingStandard(CT-Foundation)

Sun April 28, 2019 09:32 PM

Hi,

The date has been announced:

Prior to June 17, 2019, in DBR/DBR-RT, the cost of the recurring monthly RI fees (RI Fee line items) are allocated to applicable instance usage line items, and RI-related fields are transformed to present unused usage and associated costs. Starting on June 17, 2019, this functionality will be discontinued in the DBR and DBR-RT.

More info here:

https://docs.aws.amazon.com/awsaccountbilling/latest/aboutv2/detailed-billing-report-changes.html


#CostingStandard(CT-Foundation)

Tue April 02, 2019 08:19 PM

Yes, thank you Matt.  AWS is definitely planning to deprecate their original billing reports including both the Cost Allocation Report and the Detailed Billing Report.  The exact timing is still TBD and we have not heard any additional details from AWS.  We do have regular interactions with AWS, though, and will keep this group apprised of any developments or further information.  And, with the latest Multicloud Connector, we do have full support for the AWS Cost and Usage Report (CUR).

 

AWS CAR Deprecated

 

AWS DBR Deprecated


#CostingStandard(CT-Foundation)

Fri March 29, 2019 05:58 AM

Thu March 28, 2019 02:03 PM

AWS started publishing the Cost and Usage Report (CUR) last year (if not earlier). They definitely are planning to retire the Monthly Cost Allocation Report (MCAR) and Detailed Billing Report (DBR), but I'm not aware of any announcement from AWS of any specific retirement date. Given the absence of the retirement date, I expect you have a little time ahead of you to transition over to the CUR.

 

Maybe @Randall Tennant or @Apptio UserAssistance can provide some insight about anything they may have heard regarding a retirement date for the MCAR and DBR. Poke poke. 


#CostingStandard(CT-Foundation)

Thu March 28, 2019 06:11 AM

Thanks @Matt Temple We have the "Cost Allocation Report" linked and data is now flowing back into CT with some of the reports now lighting up. 

The reason I want to get the "Cost and Usage Report" up and running as well as our Cloud Lead was telling us that AWS will be retiring the "Cost Allocation Report" & "Detailed Billing Report" soon so I wanted to ensure that we could still get useful billing data flowing back into Apptio


#CostingStandard(CT-Foundation)

Tue March 26, 2019 01:44 PM

Minimally, the only AWS data source needed is the "Cost Allocation Report". It's easy enough to evolve adoption of other AWS data sources later.


#CostingStandard(CT-Foundation)

Thu March 21, 2019 06:27 AM

I found this guide very helpful when setting up our first Datalink Connector to AWS. One thing I would point out though is that there is no need to use the full address for the Role Name and S3 Bucket fields. We found that simply entering the actual Role Name and S3 Bucket name was sufficient.


#CostingStandard(CT-Foundation)

Mon November 19, 2018 01:12 PM

You can find similar instructions for Azure here: DataLink Microsoft Azure Connector Guide. Hope this helps! 


#CostingStandard(CT-Foundation)

Mon October 29, 2018 09:00 PM

This is a fantastic walk through of the AWS setup, Is there an Azure document with similar content?


#CostingStandard(CT-Foundation)