In TSCO an ETL or the TSCO Scheduler is terminating with a Program abnormally terminated [java.lang.OutOfMemoryError: GC overhead limit exceeded] error

Version 7
    Share:|

    This document contains official content from the BMC Software Knowledge Base. It is automatically updated when the knowledge article is modified.


    PRODUCT:

    TrueSight Capacity Optimization


    COMPONENT:

    Capacity Optimization


    APPLIES TO:

    TrueSight Capacity Optimization 11.3,11.0, 10.7, 10.5, 10.3, 10.0 ; BMC Capacity Optimization 9.5 ; BMC ProactiveNet Performance Management Suite



    PROBLEM:

     

    In TrueSight Capacity Optimization (TSCO), ETL execution is failing with a Program abnormally terminated [java.lang.OutOfMemoryError: GC overhead limit exceeded] error reported in the ETL logs:

      

      ERROR FATAL - Program abnormally terminated [java.lang.OutOfMemoryError: GC overhead limit exceeded]

    For example, here is the error generating during a TSCO vCenter ETL execution:

    01/26/2012 09:24:35 START [taskid=52]- Running task: 52 on scheduler 0
    01/26/2012 09:24:35 INFO [taskid=52]- Starting process /opt/cpit/etl/jetlrun.sh start 52 207
    01/26/2012 09:24:41 INFO ETL Engine starting on server [ut03262], PID=507
    01/26/2012 09:24:41 OUTPUT [LOG] /opt/cpit/etl/log/M1Q92452.out;/opt/cpit/etl/log/M1Q92452.err
    01/26/2012 09:24:41 REPORT Starting ETL in simulation mode
    01/26/2012 09:24:41 INFO ETL Engine: EXTRACT step
    01/26/2012 09:24:41 INFO DBconf: New DS_STATUS SOURCE 2
    01/26/2012 09:24:41 INFO [Insert status] inserting new status for srcid = 2
    01/26/2012 09:24:42 INFO Forcing timestamps to AS time zone: America/Chicago
    01/26/2012 09:24:42 INFO Connecting to https://sdcvvct01.ent.ad.ntrs.com/sdk
    01/26/2012 09:25:45 INFO Connected to sdcvvct01.ent.ad.ntrs.com - VMware vCenter Server 4.1.0 build-345043 UUID: 46C3168A-AC40-4EFE-9D38-0A744ED6036E - api 2.5. Supported sampling intervals: Past day [300] (interval: 5m - level: 1 - retention: 1d); Past week [1800] (interval: 30m - level: 1 - retention: 7d); Past month [7200] (interval: 2h - level: 1 - retention: 30d); Past year [86400] (interval: 1d - level: 1 - retention: 365d) diff:-21600000 - parallelism: 2
    01/26/2012 09:25:45 INFO This VirtualCenter version has a known bug in Cluster Memory Utilization performance index. ETL will fix imported values.
    01/26/2012 09:25:45 INFO Starting metric download. Forced minimum date: 25/01/2012 09:24:42. Sampling interval: Past week [1800] (interval: 30m - level: 1 - retention: 7d).
    01/26/2012 09:39:53 ERROR FATAL - Program abnormally terminated [java.lang.OutOfMemoryError: GC overhead limit exceeded]
    01/26/2012 09:39:53 INFO [Status type] setting status type to FAILED - ts = 2012-01-26 09:39:53
    01/26/2012 09:39:53 INFO [DS update status] updating datasource status for srcid = 2
    01/26/2012 09:39:56 INFO [taskid=52]- Process terminated correctly in 919.59[s]
    01/26/2012 09:39:56 STOP [taskid=52]- Ended task: 52


    For a Service Extractor ETL the ERROR FATAL - Program abnormally terminated [java.lang.OutOfMemoryError: GC overhead limit exceeded] message will generally be reported in the $CPITBASE/scheduler/log/scheduler.out or scheduler.err file.

      


    LP: TrueSight Capacity Optimization 11.0, 10.7, 10.5, 10.3, 10.0
    DR: Capacity Optimization 9.5, 9.0, 4.5


     

     


    SOLUTION:

     

    This error typically indicates that the Java Heap Size for the JVM in which the ETL is running has been exhaused and the recommended workaround is to increase the Java Heap Size of the ETL.  Note that each ETL runs in its own separate JVM which is executed by the TrueSight Capacity Optimization (TSCO) Scheduler. 

      

    This document covers increasnig the Java Heap Size associated with the two different types of ETLs:

         

    Regular ETLs run in their own separate java process -- so each separately executed ETL results in the TSCO Scheduler spawning a separate Java process in which that ETL runs.  Service Extractor ETLs run within the BCO Scheduler itself in a separate thread.

    NOTE: If the /TSCO Installation Directory]/customenvpre.sh file doesn't exist copy the customenvpre.sh.sample file to customenvpre.sh and then proceed to make the recommended changes to the customenvpre.sh described below.

      

    Increasing the ETL Heap Size for a Regular ETL

      

    The steps to increase the ETL JVM's Heap Size are:

    (1) Edit the /TSCO Installation Directory]/customenvpre.sh file on the ETL Engine where the ETL is running:

    (2) Change this:

    #ETL_HEAP_SIZE="512m"
    #export ETL_HEAP_SIZE


    To this:

    ETL_HEAP_SIZE="1024m"
    export ETL_HEAP_SIZE


    (3) Restart the Scheduler service on the machine:

    cd /[BCO Installation Directory]
    ./cpit restart scheduler


    In general in Technical Support we recommend increasing from the default of 512 MB to 1 GB and then if you continue to experience out of memory errors only then increasing beyond that to either 1536 MB or 2048 MB.  If you need to increase beyond 2 GB we would generally recommend contacting Technical Support to discuss the environment and the problem you are seeing.  This is simply because 2 GB is a common java heap size memory configuration and we have less experience in Technical Support with environments having a java heap size larger than 3 GB.

      

    Note that each running ETL runs within a separate JVM and thus when setting the Java Heap Size is it important to consider how many concurrent ETLs will be running in the environment in relation to the memory configuration of the computer.  The ETL_HEAP_SIZE setting corresponds to the Java command line 'Xmx' setting so this is the maximum heap size of the JVM not the starting heap size of the JVM.

      
    Q: Do I need to also modify the ETL_PERMGEN_SIZE setting?
      

    Typically no.  If the ETL_PERMGEN_SIZE setting is to low the ETL would fail with the following error:

      
    java.lang.OutOfMemoryError: PermGen space
      

    For the "java.lang.OutOfMemoryError: GC overhead limit exceeded" error that indicates a lack of Java Heap Space, not a lack of Java Perm Gen space.

      
    Q: Do I need to restart the BCO Scheduler after changing the ETL_HEAP_SIZE?
      

    No, regular ETLs run in a separate java process that reads the customenvpre.sh file on startup so any ETL started after the file has been updated with be executed with the increased Java Heap Size.

      
    Q: Is there any way to validate that the new ETL_HEAP_SIZE parameter has been properly set and has been picked up by the ETL?
     
      

    If the ETL_HEAP_SIZE has been correctly set when the ETL starts you'll be able to see the increased heap size on the 'java' process command line.

    So, something like this:

    > ps -ef | grep java | grep etl
    cpit     25369 25365 15 13:40 ?        00:00:01 /data1/bmc/BCO/jre/bin/java -Dsrcid=121 -Dcpit.component=etl -Xmx512m -XX:MaxPermSize=128m -Djava.awt.headless=true -Djava.io.tmpdir=/data1/bmc/BCO/temp -Dfile.encoding=UTF-8 -DETLBASE=/data1/bmc/BCO/etl -DPID=25365 ...


    The "-Xmx512m" means that the ETL_HEAP_SIZE in the custompenvpre.sh is set to 512 MB.

    After changing it to 1024 run 'ps -ef | grep java | grep etl' shows the increased heap size:

    So, updated customenvpre.sh:

    #ETL

    ETL_HEAP_SIZE="1024m"
    export ETL_HEAP_SIZE


    New 'ps -ef | grep java | grep etl' output:

    cpit     25935 25931 96 13:42 ?        00:00:04 /data1/bmc/BCO/jre/bin/java -Dsrcid=121 -Dcpit.component=etl -Xmx1024m -XX:MaxPermSize=128m -Djava.awt.headless=true -Djava.io.tmpdir=/data1/bmc/BCO/temp -Dfile.encoding=UTF-8 -DETLBASE=/data1/bmc/BCO/etl -DPID=25931...

      

    Increasing the BCO Scheduler Heap Size for a Service Extractor

      

    For a Service Extractor, such as the VMware - vCenter Extractor Service:

      

    (1) Edit the /TSCO Installation Directory]/customenvpre.sh file on the ETL Engine where the ETL is running.

      

    (2) Change this:

    #SCHEDULER_HEAP_SIZE="1024m"
    #export SCHEDULER_HEAP_SIZE


    To this:

    SCHEDULER_HEAP_SIZE="2048m"
    export SCHEDULER_HEAP_SIZE

      
    Q: Is 2048 MB of Scheduler Heap Size sufficient for all Service Extractor ETLs?
      

    No, 2048 MB is a good starting point but the the BCO Scheduler continues to terminate it may be necessary to increase the Scheduler Heap Size setting to a larger value (such as 3096 MB, 4192 MB, or for a very large Service Extractor (or a machine running multiple Service Extractors) to an even higher value.

      

    Some recommended guidelines:

      
       
    • Reserve at least 1 GB  of memory for the OS and other tasks running on the ETL Engine machine
    •  
    • Contact Technical Support if you plan to increase the Scheduler Heap Size to more than 1/2 of the total physical memory on the machine
       Related Products:  
       
    1. TrueSight Capacity Optimization
    2.  
    3. BMC Capacity Optimization
    4.  
    5. BMC ProactiveNet Performance Management Suite
       Legacy ID:KA363181

     


    Article Number:

    000095675


    Article Type:

    Solutions to a Product Problem



      Looking for additional information?    Search BMC Support  or  Browse Knowledge Articles