Discovery: How to use tw_disk_utils; Cannot access UI, Insufficient disk space to start services. The datastore files or datastore transaction logs are too large.

Version 6
    Share This:

    This document contains official content from the BMC Software Knowledge Base. It is automatically updated when the knowledge article is modified.


    PRODUCT:

    BMC Discovery


    COMPONENT:

    BMC Discovery 11.3


    APPLIES TO:

    BMC Discovery



    PROBLEM:

     

    The appliance fails to start due to insufficient disk space. The datastore files (residing either on the original disk in /usr/tideway/var/localdisk/tideway.db, or on a separate disk) have taken up the available space. This can also occur with the datastore transaction logs.  It is not possible to move/remove enough files to allow the services to start.

    How can tw_disk_utils be used from the command line to move the datastore or datastore transaction logs to a new, larger disk?

     


    SOLUTION:

     

    ************************************************************************************************************************************
    IMPORTANT NOTES: 

    1. The tw_disk_utils utility does not support LVM (see KA 000153310). The expected error message is "ERROR: Configuration error: LVM-based configuration detected"

    2. The tw_disk_utils utility can only move one set of Discovery data to a given disk. The entire new disk will be used. The data components that can be moved are:
    datastore_data
    datastore_logs
    debug_logs
    record_pool
    backup_data

    Moving a second set of Discovery data to a given disk cannot be done with tw_disk_utils, and is not recommended.

    3. The tw_disk_utils utility can move more than one set of Discovery data to different disks in the same operation. For example, to move the datastore data to /dev/sdc and the debug logs to /dev/sdd:
          tw_disk_utils --move datastore_data:/dev/sdc --move debug_logs:/dev/sdd
          
    4. The tw_disk_utils utility can create a swap partition on the same disk as part of a move operation. For example, suppose the appliance has an existing datastore partition (/mnt/addm/db_data) on /dev/sdb, and that disk also had a swap partition. The datastore has outgrown the current partition and needs to be moved to a new disk. As part of the move operation, it is recommended to also create a new swap partition on the destination disk. If this is not done in one operation, it won't be possible to add a swap partition to the new disk later.

    ************************************************************************************************************************************

    To use tw_disk_utils to move the datastore to a new, larger disk, please do the following:

    1) Stop all the Discovery services, then shut down the VM. This can be done from the command line ('sudo shutdown -h now') or simply power off the VM from the VMWare console

    2) Use the VMWare console to allocate a new disk of sufficient size for the datastore, plus growth. Do not format or partition this disk.

    3) Power the VM on after the disk is added.
     
    4) From the command line, login as user tideway and stop the services:  sudo /sbin/service tideway stop (for RedHat/CentOS 6) or tw_service_control --stop (for CentOS 7)

    5) Run fdisk ('sudo fdisk -l') to determine the device of the new disk (for example, /dev/sdb). Make a note of this name. 

      

    If new disk does not appear in the "sudo fdisk -l" results, it may be necessary to reboot the appliance and then re-check for the new disk.

    6) If the datastore currently resides on its own partition (for example /mnt/addm/db_data on /dev/sdb), do the following to determine if that disk also has a swap partition:

        Run "swapon -s". If the output looks like this:
        
        Filename                                Type            Size    Used    Priority
        /dev/sda2                               partition       8388604 2581100 -1
        /dev/sdb2                               partition       8388604 1234500 -2
        
        Then an 8gb swap partition exists on /dev/sdb, and a similar partition should be created on the new disk. See below for further steps in this case.

    7) Change the working directory to /usr/tideway:
        cd /usr/tideway

    8) Move the datastore to the new disk with the following commands:

        screen
        
    The screen command can restore a terminal session if the connection is dropped. See https://docs.bmc.com/docs/display/DISCO113/Using+screen.

    To move the datastore from its default allocation (/dev/sda) to a new, separate datastore partition, the command could look like this:

        tw_disk_utils --move datastore_data:/dev/sdb   

    Replace /dev/sdb with the correct name of the new disk noted in step 5 above. This command will format and partition the new disk and move the datastore to it. 

    Note: if moving the datastore transaction logs, use "datastore_logs" instead of "datastore_data" in the command above.

    To move from an existing separate datastore partition and also create a swap partition, the command could look like this:

        tw_disk_utils --move datastore_data:/dev/sdc  --new-swap 8000:/dev/sdc
        
    Replace /dev/sdc with the correct name of the new disk noted in step 5 above.

    NOTE:  Some errors are normal and can be ignored, such as the one below.  If provided with the option to proceed in spite of errors, do so.
    [tideway@appliance ~]$ tw_disk_utils --move datastore_data:/dev/sdc --new-swap 8000:/dev/sdc
    Fri Apr 17 13:57:53 2020 : tw_disk_utils started.
    Retrieving the system configuration
    ERROR: sudo parted print command error: Error: /dev/sdc: unrecognised disk label
    Validating the system configuration
    Checking /etc/fstab: all entries must be valid
    Verifying that there is enough space in the new disks
    Validating the move requests
    Estimating the overall process duration
    WARNING: tw_disk_utils will shut down BMC Discovery services
    The disks that will host the new partitions will be completely
    formatted (all data stored in those disks will be deleted)
    INFO: rough estimated duration: 9 hours
    Do you wish to proceed? (y/n) y


    If the terminal session is lost while tw_disk_utils is running, it can be reattached with "screen -r".

    If "screen -r" does not restore the session (or screen was not run), then the log can be followed like this:   tail -f /usr/tideway/log/tw_disk_utils.log


    9) After the tw_disk_utils is finished, it will say "Completed". At that point, the tideway services can be restarted, and the UI should be available.

    Use tw_disk_utils --help  to see help for the other features.

    Please also see the following video "How to resolve disk space problems in BMC Discovery".

     


    Article Number:

    000324095


    Article Type:

    Solutions to a Product Problem



      Looking for additional information?    Search BMC Support  or  Browse Knowledge Articles