AWS Cost and Usage Extractor ETL - Throbuleshooting

Version 1

    This document contains official content from the BMC Software Knowledge Base. It is automatically updated when the knowledge article is modified.


    TrueSight Cloud Cost Control


    Capacity Optimization


    11.0.00, 11.3.01


    I am a TrueSight Cloud Control user and I set up the Amazon Web Services - AWS Cost and Usage Extractor, but in CCC View I have only on-prem costs and nothing related to AWS.

    Which is the basic steps I can do to check if I setup the ETL properly? 



    1. ETL logs Manifest file does not exist


    In the ETL logs there lines like this one
    2018-05-29 15:46:29 INFO Manifest file: /my-report-name/20180301-20180401/my-report-name-Manifest.json does not exist for the month: 20180301-20180401 in the S3 bucket: my-s3-bucked-name


    The manifest.json file is produced/updated internally by AWS and contains references to the path of the generated cost report
    AWS produces the cost report and create/updated the manifest file once a day.
    After the creation of the report on AWS, it is necessary wait that this report is generated at least one time: if the ETL run immediatly after the configuration of the report, it is normal that the manifest file doesn't exist. We suggest to wait one day before run the ETL for the first time after the creation of report on AWS

    Also, when the ETL run for the first time, it automatically recover last 6 months and so it is normal if this kind of messages.
    E.g.: The report has been created on AWR on 25st of May 2018 and the ETL run for the first time on 30 of May. The ETL start to import cost reports from 1st of January, but these reports are not present and so it is normal see that kind of message in the log


    2. Check the ETL Configuration (Report name and report Prefix)

    During the creation of the Cost Usage report in AWS, it is required to specify a report name (mandatory) and optionally a Report path prefix.                            
    User-added imageUser-added image
    These two parameter must be used in the ETL configuration 
    User-added image  

    3. HTTP 403 error accessing to report

    If there is an error HTTP 403 in the ETL log when it tries to download the MANIFEST, please check the permission of the AWS user used by the ETL and check if can access in read mode to the report  

    4. Cloud Cost Control View - By Server page does not contain AWS machine

    On AWS, in   AWS Cost and Usage report configuration, check if:  
    1. Time unit is set to Daily
    3. Include Resource ID" option is enalbed
       User-added image
    If   Resource Ids is NOT checked:  
    • AWS Cost and Usage Extractor does not have any reference of the id of the resource
    • AWS Cost and Usage Extractor generates fake resource id beased on the bill line item id
    How to easily check if the resource ids has been enabled:  
    • Go to By Cloud Service page in CCC View
    • Drill down to Amazon EC2
    • Go to tab Resources
    • Add the resource id column to the Resources table with the Show/Hide Columns
    • Check if the resource id of a VM Instance starts with i-
           767138742567-AmazonEC2-Tax  i -Oad809ebd7f7a30d3  i•62d6acdd  Tax for product code ArmazonEC2  panama-a r  65589  605.38  564.19  VM Instance  VM Instance    

    5. ETLs Entity Catalog

    Configuration and utilization metrics for AWS system are imported by the   AWS API Extractor ETL. 
    Make sure the two ETLs are using the same shared entity catalog 


    Article Number:


    Article Type:


      Looking for additional information?    Search BMC Support  or  Browse Knowledge Articles