Discovery: Appliance is unable to start because the /usr partition is full. What files can be removed safely (as a temporary solution) so that the services can start?

Version 3
    Share This:

    This document contains official content from the BMC Software Knowledge Base. It is automatically updated when the knowledge article is modified.


    PRODUCT:

    BMC Discovery


    COMPONENT:

    BMC Discovery 11.3


    APPLIES TO:

    BMC Discovery 11.3



    PROBLEM:

     

    The appliance services cannot start because the /usr partition is full. What files can be removed safely (as a temporary solution) so that the services can start?

    Para una version en Español de este articulo ver KA 000191371

     


    SOLUTION:

     

    ************************************************************************************************************
    IMPORTANT: REGARDING DATASTORE TRANSACTION LOGS:

    It is possible that the datastore transaction logs are filling up the /usr partition. These files should never be deleted. Moving or deleting these files will damage the datastore, likely beyond recovery.

    The doc at https://docs.bmc.com/docs/discovery/113/limitations-and-restrictions-of-this-version-788109971.html says:

    >>>>>>
    Failure to comply with the following instructions will result in datastore corruption, and in some cases unrecoverable datastore corruption.

    Modification of datastore files and logs
    Under no circumstances should you add, remove, or amend any of the datastore files or datastore log files without explicit clearance from BMC Customer Support.

    The following are examples of datastore file names:
    pa55bc128f62ce9c427a1d742_nHost_hidx
    pa55bc128f62ce9c427a1d742_nHost_hist
    __db.001
    DB_CONFIG
    main

    The following are examples of datastore log file names:
    log.000002301
    log.000002302

    The location of the datastore and the datastore log files can be obtained by reading the /usr/tideway/etc/link.conf file. Under no circumstances should you modify this file once a system has been commissioned.
    <<<<<<


    Run these commands to check the size of the datastore transaction logs:

    cd /usr/tideway/var/tideway.db/logs
    du -sh

    If these files are filling up the /usr partition, contact Customer Support before taking any other action.
    ************************************************************************************************************

    First, determine what are the largest files in /usr by running this command:
    du /usr/tideway | sort -nr | head -n 30

    Before removing any files, make sure all the services are stopped:
    (RedHat) sudo /sbin/service tideway stop
    (CentOS) tw_service_control --stop

    These files can be safely removed:

    1) Record and pool data:
    cd /usr/tideway/var/record
    rm -rf *
    cd /usr/tideway/var/pool
    rm -rf *

      

    2) Data that was gathered for support:
    cd /usr/tideway/var/gather
    rm *

      

    3) Discovery log files (*only* those in /usr/tideway/log directory!). 

    Try to remove the oldest ones first, as newer files could be important for troubleshooting purposes. Good candidates for removal are files that have already been compressed, as these are older files. To remove all of these:

    rm -f /usr/tideway/log/*.gz

    Then list the remaining files by oldest first to find additional candidates"

    ls -ltr /usr/tideway/log | more

    If there are really large files that need to be deleted (as shown in the output of the 'du' command above), these can be removed also. However, do not remove files that have been modified on the current day. 
     

      

    4) Check for core files in /usr/tideway/cores. 
    cd /usr/tideway/cores
    ls -la 
    ## Note: If core files exist, keep a copy of the listing (from the "ls -la" command.
    ## Prior to deleting a core file, check to see what service created it by running:
    file <core file name>
    ## If possible, contact Customer Support before deleting the core files. Also, save a copy of any *.txt files in a backup directory
    ## Run this command to remove the core files:
    rm -f core*

    5) Any local backups that are no longer needed. For example, if a newer backup has been created on a remote server.
    cd /usr/tideway/var/backup
    ls -la
    rm -f *

    6) Check for large "pq" files, but don't delete them.
    cd /usr/tideway/var/persist/reasoning/engine/queue
    ls -laSh | more
    <see the largest files at the top of the screen>
    Contact Support before deleting these files.

      

    After removing files, run 'df -h' to ensure that > 2gb of space is now available in /usr. Then restart the services:
    (RedHat) sudo /sbin/service tideway start
    (CentOS) tw_service_control --start

      


    NOTE:  These are only temporary measures.  To fix the problem permanently, it's important to determine what caused the filesystem to become full, and why.

    See these related Knowledge Articles:
    000094102 - The appliance has shut down because available disk space is low
    000028149 - Core file(s) have been created in /usr/tideway/cores. Why are these files created, and what should be done before deleting them?
    000082640 - How to use tw_disk_utils; Cannot access UI, Insufficient disk space to start services, Datastore files are too large.

    Please also see the following video "How to resolve disk space problems in BMC Discovery".

     


    Article Number:

    000035760


    Article Type:

    Solutions to a Product Problem



      Looking for additional information?    Search BMC Support  or  Browse Knowledge Articles