Discovery: The appliance has shut down because available disk space is low. The appliance services cannot be restarted.

Version 3
    Share This:

    This document contains official content from the BMC Software Knowledge Base. It is automatically updated when the knowledge article is modified.


    PRODUCT:

    BMC Discovery


    COMPONENT:

    BMC Discovery 11.3


    APPLIES TO:

    BMC Discovery



    PROBLEM:

     

      The appliance has shut down because available disk space is low. It is not possible to restart the appliance at this point. 

     


    SOLUTION:

     

    The first step is to determine which partition is full. This can be done by running "df -h" from the command line. Typically, either /usr or the datastore partition is full.

    The output of 'df -h' may not show any partition at 100% capacity, but whichever one reports the highest percentage usage is likely the one that is having the problem.

    If a Support case will be opened, please always send the results of "df -h".

    To prevent problems that can occur if the disk partition containing the datastore files or datastore transaction logs runs out of space, the free disk space monitor is used. See  https://docs.bmc.com/docs/display/disco113/Configuring+disk+space+monitor.  If there is less than a predefined amount of space on the disk, the monitor shuts down Discovery gracefully, and/or prevents it from starting. By default the minimum amount of space permitted on the disk is 1024MB.

      
      Potential root causes of saturation of the datastore partition: 
    - The datastore is simply too big, the appliance has not been correctly sized (see https://docs.bmc.com/docs/display/disco113/Sizing+guidelines)  
     
    - A defect may increase the datastore size. See KA    000123599 to check this root cause and resolve the problem.  
     
    - A problem may occur with the deletion of doomed nodes (See   KA    000093640). Note that this condition can be aggravated when compaction is not run on a regular basis. To confirm this, resolve the doomed node   
    accumulation issue (using the mentioned KA), then do a compact, and compare the entries in the tw_appliance_df.log file before/after the resolution of the doomed nodes issue. If the datastore growth rate decreases afterwards, this means that a regular compaction is needed to avoid this cycle (large datastore => performance issue => doomed nodes accumulation => large datastore => performance issue...).  
     
    - Defect DRUD1-16602 causes an accumulation of relationships in the history. This is fixed in Discovery 11.1, 11.0.0.3, 10.2.0.6,  and 10.1.0.5 releases.   
     
    The solution is to add a new, larger disk to the appliance and move the datastore to it. For more information, see KA    000082640  . 
     
    A best practice is to compact the datastore on a regular basis.  
     
      Potential root causes of saturation of the datastore log partition: 
     
    - The "Time before history entries are purged" setting is lowered (for example, from "Never" to 30 days). This can result in a large number of history nodes to suddenly be eligible to be purged, which in turn causes a spike in the size of the datastore transaction logs. For resolution, contact Customer Support and reference internal KA #000164304.  
     
    - In theory, a performance issue could slow down the datastore, lead to an accumulation of datastore transactions pending in the datastore log partition and then a saturation of this partition. This cause-effect link has never been confirmed so far, this is a suspected root cause.  
     
      Potential root causes of a /usr saturation:
      

    - The /usr/tideway/cores directory contains core dump files (these can be very large). Follow the steps documented in KA 000028149. Afterwards, the contents of the folder can be deleted, but keep the folder itself.

    - The /usr/tideway/log directory is full of log files that contain debug-level information. The largest files can be compressed to free up space. Make sure "debug" level is only used when needed.

    - There are too many (or too large) reasoning transaction files in the /usr/tideway/var/persist/reasoning/engine/queue directory. A large amount of these files may be kept during the time scans are on hold. This is normal, they should be processed and deleted when the scan restarts. Deleting pending transaction files should only be done at the direction of Customer Support. In rare cases, deleting other files (such as files in /usr/tideway/log) can allow the appliance to restart and process the pending .pq files. 

    Note that attempting to mass destroy a large number of hosts (>1000) from the UI could create a very large .pq file. Depending on the available space in /usr, this could lead to a disk saturation. It is recommended to destroy hosts in smaller batches (<1000) and monitor the disk space when doing this action. With version 11.3.01 and later, an alternative is to use "tw_query --destroy". See https://docs.bmc.com/docs/display/DISCO113/tw_query

    - For a consolidator, there are too many files in the /usr/tideway/var/persist/consolidation directory. This could happen if the consolidator was unavailable for an extended time while the scanners continued to process Discovery runs. See KA 000118676.

    - The local backup takes too much space in /usr/tideway/var/localdisk/backup. If desired, the content can be moved to another location such as a remote SSH server or a Windows share. Then delete the content by running 'rm -f /usr/tideway/var/localdisk/backup/* ', keeping the folder itself. If this was the root cause of the disk saturation, the possible solutions are to:
    > Use the UI disk utility to add a new disk, and move the backup function to it. See https://docs.bmc.com/docs/display/disco113/Managing+disks+and+swap+space.
    > Create the next backup on a remote SSH or Windows server. See https://docs.bmc.com/docs/display/disco113/Backing+up+and+restoring+the+appliance.

    - There is too much recorded data in /usr/tideway/var/pool and /usr/tideway/var/record. The contents of these folders can be deleted, but not the folders themselves. Make sure that record mode is not enabled (see "Recording Mode" in https://docs.bmc.com/docs/display/disco113/Configuring+discovery+settings

    - If the datastore (or datastore transaction logs) is stored in /usr, see the section "Potential root causes of saturation of the datastore partition" above.

      
      You can identify the root cause with commands like:  
    df -h   
    du -h /usr/tideway | grep [0-9]G  
     
    Please also see the following video "How to resolve disk space problems in BMC Discovery".  
     

     


    Article Number:

    000094102


    Article Type:

    Solutions to a Product Problem



      Looking for additional information?    Search BMC Support  or  Browse Knowledge Articles