In TSCO, what is the best practice way to recover data processing from a failed AutoETL import that updated the Last Counter values?

Version 3
    Share This:

    This document contains official content from the BMC Software Knowledge Base. It is automatically updated when the knowledge article is modified.


    PRODUCT:

    TrueSight Capacity Optimization


    COMPONENT:

    Capacity Optimization


    APPLIES TO:

    TrueSight Capacity Optimization 11.3, 11.0, 10.7, 10.5, 10.3, 10.0



    QUESTION:

    In TrueSight Capacity Optimization (TSCO) the Gateway Server vis parser ETL configured to obtain the Visualizer files via General Manager ran but during the LOAD phase the SQL*Loader failed due to a database connectivity problem.

    This resulted in errors like this being reported by the Gateway Server vis parser ETL:

    04/29/2015 06:03:21 INFO Executing SQLLOADER. ControlFile=/sys_apps_01/cpit/etl/output/SYSLPARGLBP4T5Z1109239.ctl LogFile=/sys_apps_01/cpit/etl/output/SYSLPARGLBP4T5Z1109239.ctl.log
    04/29/2015 06:03:21 FAILED BCO_ETL_FAIL107: SQL*Loader-70x: ORACLE SQL*Loader fatal error occurred. Check database availability and instane status.
    04/29/2015 06:03:21 WARNING Loader - dataset SYSLPARGLB - Rows in dataset: 4144
    04/29/2015 06:03:21 WARNING Loader - dataset SYSLPARGLB - Rows read from ctl_file: -1
    04/29/2015 06:03:21 WARNING Loader - dataset SYSLPARGLB - Rows loaded in Oracle: -1
    04/29/2015 06:03:22 FAILED BCO_ETL_ERR117: Loader - dataset SYSLPARGLB - [ORA-12520] error count: 1
    04/29/2015 06:03:22 FAILED Load completed with exit code 256. Check /sys_apps_01/cpit/etl/output/SYSLPARGLBP4T5Z1109239.ctl.log for additional details.
    04/29/2015 06:03:22 FAILED Errors are not data-dependent. Check database status.


    This resulted in the "Last Counter" values being incremented in the ETL configuration.

    Applies To

    TrueSight Capacity Optimization 11.3, 11.0, 10.7, 10.5, 10.3, 10.0


     


    ANSWER:

     

    There are two paths available (with the second path having two variants) for the recovery of the failed import.
      
    Options: 

      
       
    1. For each of your AutoETL generated vis parser ETLs update the Last Counter back to the beginning of the interval that needs to be recovered. 
    2.  
    3. Create one or more recovery ETLs that will import Visualizer files from a defined input directory 
      

    Option A: For each of your AutoETL generated vis parser ETLs update the Last Counter back to the beginning of the interval that needs to be recovered. 

      

    This is an option if:

      
       
    • The Last Counter has been updated but the data hasn’t really been successfully imported into CO (for example, due to an error within the SQL*Loader itself 
    •  
    • Data hasn’t been imported from the current date back a day or two. 
      

    This isn’t an option if:

      
       
    •  You are planning to import data from a previous date range before the current date (for example, you want to recover data from 3/16 - 3/17 and it is currently 3/19. 
      

    Also, think about how long the ETL chain is going to run.  So, if your ETL chain executes at 5 AM and usually runs for 6 hours to import one day of data and the recovery is being started at 8 PM to recover 2 days of data, then the recovery run is going to impact the nightly ETL execution (because the chain will still be running at 5 AM when the nightly ETL is supposed to start importing data).  But, if you are executing the recovery at 10 AM, then the chain will be finished around 10 PM (6 hours * 2 days) so it will be done long before the next ETL execution is scheduled to begin. 
      
    To do this: 

    (1) Edit each of your existing GatewayServer_[console]_#_# vis parser ETLs and click the ‘Last Counter’ link at the top.  This will bring up the “Lastcounter Status Detail” screen.

    (2) Click the ‘Edit lastcounter’ button.

    (3) In the ‘Lastcounter’ field change the current Lastcounter to the beginning of the date range for recovery.
     
    For example, change it from “2015-04-29 23:59:00” to “2015-04-27 23:59:00” to recover the data from 4/28 and 4/29. 

    (4) Click the ‘Save’ button to save the change.

    (5) Repeat for all ETLs and then manually execute the chain.

    As long as the time range to be recovered is limited (2 days or less) and is data from the previous days leading up to the current date this would be a good option. 

      

    Option B: Create one or more recovery ETLs that will import Visualizer files from a defined input directory (excluding relationship data)

      

    This option is more flexible, but does require the Visualizer files to be recovered to be manually copied from the Gateway Server to a directory on an ETL Engine (or Engines) where the recovery ETL will be configured to run. 
      
    This option can be used to recovery any historic date range of data.  It should probably be limited to recovering only one or two days of data at a time -- although if multiple ETLs are being created to execute the recovery across multiple ETL Engine Servers it may be possible to recover more days in a single run. 

    This option will disable the import of the Object Relationship data for the recovery run (basically meaning that the previously defined relationships from the last ETL execution will persist.  This is generally appropriate for most environments but if the TSCO Gateway Server ETL is importing data from highly dynamic virtualized environments it may be necessary to use 'Option C' instead.
      
    To implement this option: 
      
    (1)    Create a new ETL called “Gateway Server [console] Recovery #” (or really whatever you want). (Administration -> ETL & System Tasks -> ETL Tasks -> Add -> Add ETL) where ‘[console]’ is the Gateway Manager that this ETL is going to recover imported data and “#” is the number of the run.  For example, if my nightly ETL chain is called GatewayServer_hostname.domain.com_0_0 then I might call this “GatewayServer_hostname.domain.com_ Recovery_1”.  But the exact name you pick really doesn’t matter. 
      
    (2)    Within the ETL configuration:

           a.      Set the ETL Module to be “BMC - TrueSight Capacity Optimization Gateway VIS file parser”

           b.      Select all the desired ‘Platforms’ (usually it would be appropriate to select all platforms)

           c.      Select the desired Data types (this should generally match the data types selected for the regular daily ETL)

           d.      Click the 'Edit' button in the Datasets section and remove '[59] OBJREL - Object relationships' from the 'Selected datasets' list.  Click 'Apply' in the Datasets section.  This instructs this ETL to not import any object relationship data which means that any host to virtualization guest (for example, AIX frame to AIX partition) will not be updated by this recovery execution.  For most environments this is acceptable because the parent to child relationships for TSCO Gateway Server entities will be relatively static.  But, if your environment is highly dynamic see 'Option C' below which uses an alternate ETL configuration to import the objection relationship data.

           e.      Under the ‘Entity catalog’ heading, select “Shared Entity Catalog” and select the Entity Catalog used by the nightly BPA Vis parser ETL. It is important to select the right catalog, verify this by open the ETL from Administration >ETL & SYSTEM TASKS >ETL tasks which is processing the data on regular schedule  and click on the Entity Catalog beside the LastCounter and Edit options

           f.      Under the ‘Vis Parser Configuration’ heading select ‘via file’.

           g.      Under the ‘File location heading’:

                  i.      Select ‘Local directory’

                  ii.     Set the ‘Directory’ parameter to a directory on the local ETL engine where the vis files to be recovered resides

                  iii.    Set ‘Recurse into subdirs?’ to ‘No’

                  iv.     Set ‘After parse operation’ to ‘Archive parsed file in directory’

                  v.      Set ‘Archive directory (local)’ to a local directory that exists on the ETL Engine Server or use the default “%BASE/../repository/imprepository” value

           h.       Click the ‘Advanced’ button at the bottom of the window.

           i.       Click the ‘Apply’ button to apply the dataset change

           j.       Click the ‘Save’ button at the bottom of the screen.
      
    (3)    Now that the ETL have been created and saved, you can separately copy the input Visualizer files into the directory on the ETL Engine that was specified in the ETL as the input files directory. 
     
    (4)    This process could be repeated to have separate recovery ETLs running on different ETL Engine Servers each handing a subset of the Visualizer files to be recovered.  Each recovery ETL would need to have its own separate input directory. 
      
    It may be possible to implement something more clever with these recovery ETLs in the future.  For example, the General Manager Lite functionality knows which Visualizer files have computers that weren’t successfully imported into BCO.  That list could be parsed by a custom script and the Visualizer files could be copied to a set of directories that could be used as the input directory for the recovery vis parser.  If the vis files were in a specific source directory on the Gateway Server for the recovery ETL we could even configure the vis parser recovery ETL on the CO ETL Engine side to use the ‘scp’ or ‘sftp’ transfer method and just have it transfer all of the Visualizer files from the input directory automatically. 

      

    Option C: Create one or more recovery ETLs that will import Visualizer files from a defined input directory (including relationship data)

      

    This option is basically the same as option B but will import the Object Relationship data so any parent and child relationships that changed during the recovery period will be visible in TSCO.  This path is really only necessary when TSCO is importing highly dynamic virtualized environment data.  For my TSCO Gateway Server data imports option B where the object relationship data is excluded from the recovery import is a cleaner solution.
      
    To implement this option: 
      
    (1)    Create a new ETL called “Gateway Server [console] Recovery #” (or really whatever you want). (Administration -> ETL & System Tasks -> ETL Tasks -> Add -> Add ETL) where ‘[console]’ is the Gateway Manager that this ETL is going to recover imported data and “#” is the number of the run.  For example, if my nightly ETL chain is called GatewayServer_hostname.domain.com_0_0 then I might call this “GatewayServer_hostname.domain.com_ Recovery_1”.  But the exact name you pick really doesn’t matter. 
      
    (2)    Within the ETL configuration:

           a.      Set the ETL Module to be “BMC - TrueSight Capacity Optimization Gateway VIS file parser”

           b.      Select all the desired ‘Platforms’ (usually it would be appropriate to select all platforms)

           c.      Select the desired Data types (this should generally match the data types selected for the regular daily ETL)

           d.      Under the ‘Entity catalog’ heading, select “Shared Entity Catalog” and select the Entity Catalog used by the nightly BPA Vis parser ETL. It is important to select the right catalog, verify this by open the ETL from Administration >ETL & SYSTEM TASKS >ETL tasks which is processing the data on regular schedule  and click on the Entity Catalog beside the LastCounter and Edit options

          e.      Select as the Destination Domain the same domain as used for the original domain. In versions after 10.x Enrich domain tree and ETL Migration, it is important to select Enrich domain

           f.      Under the ‘Vis Parser Configuration’ heading select ‘via file’.

           g.      Under the ‘File location heading’:

                  i.      Select ‘Local directory’

                  ii.     Set the ‘Directory’ parameter to a directory on the local ETL engine where the vis files to be recovered resides

                  iii.    Set ‘Recurse into subdirs?’ to ‘No’

                  iv.     Set ‘After parse operation’ to ‘Archive parsed file in directory’

                  v.      Set ‘Archive directory (local)’ to a local directory that exists on the ETL Engine Server or use the default “%BASE/../repository/imprepository” value

           h.       Click the ‘Advanced’ button at the bottom of the window.

           i.       Set the "Data recovery mode" option to "True".  This setting configures the ETL to not remove entities from Workspace domain tree but still build the relationships between imported entities and any child entities also being imported

           j.       Click the ‘Apply’ button to apply the dataset change

           k.       Click the ‘Save’ button at the bottom of the screen.
      
    (3)    Now that the ETL have been created and saved, you can separately copy the input Visualizer files into the directory on the ETL Engine that was specified in the ETL as the input files directory. 
      
    (4)    This process could be repeated to have separate recovery ETLs running on different ETL Engine Servers each handing a subset of the Visualizer files to be recovered.  Each recovery ETL would need to have its own separate input directory. 

       Related Products:  
       
    1. TrueSight Capacity Optimization
    2.  
    3. BMC Capacity Optimization
       Legacy ID:KA426373

     


    Article Number:

    000030553


    Article Type:

    FAQ/Procedural



      Looking for additional information?    Search BMC Support  or  Browse Knowledge Articles