In TSCO, Historical recovery is missing data. How to perform an historical recovery

Version 6
    Share This:

    This document contains official content from the BMC Software Knowledge Base. It is automatically updated when the knowledge article is modified.


    PRODUCT:

    TrueSight Capacity Optimization


    COMPONENT:

    Capacity Optimization


    APPLIES TO:

    All CO versions



    PROBLEM:

     

    In TrueSight Capacity Optimization (TSCO) I tried to perform a historic recovery of data but either none of the historic data was imported into TSCO or only a subset of the data (the most recent data) was successfully imported into TSCO.  What is the correct way to import historic data into TSCO?  What would limit TSCO from successfully importing data older than say, 30 days, from a historic data run?

    TrueSight Capacity Optimization 11.x, 10.x

     


    SOLUTION:

     

    For TSCO version 10.3 and later

    From TSCO 10.3, it is not required to direct change the aging configuration.

    You can simply temporary disable the aging configuration going in page  Administration -> Data Warehouse -> Aging configuration and click the Aging State button.  

    User-added image


    Once the Aging configuration is disabled, you should see this:

    User-added image


    You can find the procedure in official documentation.

    The default Detail level aging configuration is 30 days and thus if an ETL is going to import data older than 30 days it is necessary to set the 'Aging state' to disabled.  That will allow the Datahub to import data that is older than the Detail level aging configuration (with Aging enabled any data beyond the Detail aging will be dropped by the Datahub and not imported).

    In general if the ETL logs indicate that data is being successfully imported but that data isn't visible in TSCO the most common reason is that the data was older than the defined Detail data retention period:

    So for example, these lines in the ETL log (in particular the LOADEDTS values) indicate that the ETL loaded performance data to the staging area to be processed by the TSCO Datahub for import ito TSCO:

    [2017-11-10 00:01:18] STAT  #NEWENTITIES=0
    [2017-11-10 00:01:18] STAT  #LOADEDCONF=1005
    [2017-11-10 00:01:18] STAT  #LOADEDTS=603
    [2017-11-10 00:01:18] STAT  #LOADEDOBJREL=402
    [2017-11-10 00:01:18] STAT  #ELAPSEDTOTAL=75
    [2017-11-10 00:01:18] STAT  #ELAPSEDEXTRACTION=21
    [2017-11-10 00:01:18] STAT  #ELAPSEDLOAD=54
    [2017-11-10 00:01:18] STAT  #LOADEDENTITIES=200

     

      

    NOTE:

      

    Once the Aging configuration is disabled on the Data warehouse  you will need to wait until the ETL that imports the aged data finishes. Then make sure to let the store-wkld-launcher wakes up and processes the data accordingly. To ensure the data is loaded appropriately we recommend these steps:

    1. Disable Aging
    2. Run ETL and wait until it finishes
    3. Tail $CPITBASE/datahub/log/wh.log and check until [store-sys-launcher] Wakes up. This component could be different if you are importing business driver data, in that case it would be [store-wkld-launcher] 

      

    3.1. Check any message that says * [store-sys-launcher] Sleeping...*

      

    2019-12-04 00:02:48,182 INFO  - [wh-sys-launcher] Woke up
    2019-12-04 00:02:48,198 INFO  - [wh-sys-launcher] executions=1, running=0, parallelism=8.0, elapsed=0.015[s], objDelta=1, objThroughput=66.66667[objs/s], procDelta=1, procThroughput=66.66667[proc/s]
    2019-12-04 00:02:48,198 INFO  - [wh-sys-launcher] Sleeping...

     

      

    4. See if data is available. If yes, enable Aging again.


    For TSCO versions before 10.3

      

    To perform an historical recovery in CO data, if the data to recovery is older than the aging of detail table (default is 15 days), follow these steps:

      

     Go in page ADMINISTRATION > DATA WAREHOUSE > Aging configuration

      

      

    Edit the configuration of SYS_DATA_DETAIL TABLE

      

      

    For more details about aging, please refer to official product documentation

      

    Go in page ADMINISTRATION > DATA HUB > Status > "Core Services - Near-real-time Warehouse service"

      

      

    Reload the configuration of warehouse, clicking on "Reload configuration"

      

      
       
    1. Enlarge the aging of SYS_DATA_DETAIL table (WKLD_DATA_DETAIL if you want recovery Business Drivers data), for examples 365 days.
    2.  
    3. Reload the configuration of Near-real-time Warehouse. 
    4.  
    5. You need to move back the last counter of the ETL in order to start the import back in  the time

      To change the last counter, you have to go in the ETL page, in the status page, click on Edit last counter

      Inside the lastcounter editor page you have to change the date for one resource and copy this date in all the other resources using the copy button

      If you don't have the lastcounter already set, some ETLs have another property named Default last counter. For example to do a historic import of data via the BPA CDB ETL you can set the 'Default last counter' parameter (which is available in Run configuration tab under the Advanced settings, see below) to the beginning of the time range you want to import.
    6.  
    7. For some ETLs, there is a property to set the max days for extraction. This screenshot shows you, as an example, the BPA CDB Extractor. Read the KB related to ETL module you are using for recovery to check if there is this properly to set. If you need to run a long recovery (i.e: 6 months), keep this number low (ie 7/15 days) and run the ETL more than once.

       

    8.  
    9. Launch the ETL manually and wait for the end of the execution
    10.  
    11. Wait for the end of data warehousing process: you can check current status looking the queues inside ADMINISTRATION > DATA HUB > Status > Core Services - Near-real-time Warehouse service > Show queues status
      


    After the historical recovery of data has been completed, restore the aging of the detail table at the starting value.

    *** NOTE some ETL's may need the data resolution to allow the ETL to import the data. For example SCOM data can change from raw to hourly as the data is aged in the SCOM DWH.

    Legacy ID:KA350735

     


    Article Number:

    000097275


    Article Type:

    Solutions to a Product Problem



      Looking for additional information?    Search BMC Support  or  Browse Knowledge Articles