Discovery: Upgrade fails: FATAL: Failed to upgrade the Operating System, Insufficient disk space, No space left on device

Version 3
    Share This:

    This document contains official content from the BMC Software Knowledge Base. It is automatically updated when the knowledge article is modified.


    PRODUCT:

    BMC Discovery


    COMPONENT:

    BMC Discovery 11.2


    APPLIES TO:

    BMC Discovery



    PROBLEM:

     

      The tw_run_upgrade.log reports errors such as : 
      
     
    misc.upgrade.display: INFO: [status display msg] 1.1: Upgrading the Operating System[FAILED] 
      
      misc.upgrade.display: INFO: [status display msg]    FATAL: Failed to upgrade the Operating System.
      
      misc.upgrade.display: INFO: [status display msg] See /usr/tideway/log/upgrade_xx.x.log for more information 
      
      misc.upgrade.display: INFO: [status display msg]    Failed to run "sh /usr/tideway/var/upgrade/extract/ADDM_Upgrade_xx.x_xxxxxx_ga.sh --managed-upgrade --auto": return code 1
      
      misc.upgrade.display: ERROR: Failed to run "sh /usr/tideway/var/upgrade/extract/ADDM_Upgrade_xx.x_xxxxxx_ga.sh --managed-upgrade --auto": return code 1
      
        
      
      The upgrade_xx.x.log may contain errors like this:  
      Extract directory on the same disk as the datastore - need extra 1GB space.
      
           Insufficient disk space to extract. 512MB required but only 16MB available in directory: /var - exiting. 
      
      or 
      
           IOError: [Errno 28] No space left on device (when the saturated partition is /usr) 
      
        
      
      On other cluster members (where the upgrade worked) the following error may be seen:   
      
      cluster_manager.clustermanager: CRITICAL: Cluster startup checks failed: Incompatible product versions 
      
        
      
      Reboot does not help. It's not possible to access the UI for the appliance affected by this issue.    
      

     


    SOLUTION:

     

    Legacy ID:KA428404

      
      Cause: Disk space saturation during the upgrade.   
      
        
      
      Solution 1:
      
      - Correct the disk space issue for the filesystem named in the log entries. Delete or move files as necessary to free up disk space.  
    - Verify that the compressed upgrade archive has been downloaded from EPD and uploaded to the /usr/tideway/var/upgrade directory of the appliance on which the upgrade is to be run. For a cluster, this can be any cluster member. To upload the upgrade archive using the UI, there must be sufficient space in /tmp, approximately 1.6 GB. 
      
      - Verify that there is at least 2GB free on /usr after having uploaded the upgrade file. If for some reason this is not possible, use the option --tmpdir option as documented in    https://docs.bmc.com/docs/discovery/113/tw_run_upgrade-788110385.html .  
    Extract: "If a machine is likely to run out of disk space, you can set a symlink for the upgrade directory to a different mounted file system which has free disk space." 
      
      - Execute the following command as root : 
      
      tw_run_upgrade --restart 
      
        
      
      Solution 2:
      
      Use this only if solution 1 is not enough to resolve the problem. 
      
      - As root, run ADDM_Upgrade_<Version>.sh --managed-upgrade-checks 
      
      - As root, run ADDM_Upgrade_<Version>.sh --managed-upgrade --auto --password <password> 
      
        If a vault passphrase has been defined, add --passphrase <passphrase> 
      
        
      
      Once the upgrade is finished, the UI may show the end of the upgrade, but still not allow a login to the UI. The upgrade logs are clean and the other cluster members report that the health of this member is ok. If this occurs, see KA    000072346. Extract:
       
      Cause: A lock file like /usr/tideway/var/cluster-manager-redirect-marker has not been automatically removed after the restart.
       Workaround: Delete the lock file and try to login again. It may be necessary to reboot after deleting the file.

    References:
    https://docs.bmc.com/docs/discovery/113/preparing-for-upgrade-788110104.html
    https://docs.bmc.com/docs/discovery/113/upgrading-bmc-discovery-788110106.html
    https://docs.bmc.com/docs/discovery/113/troubleshooting-upgrade-issues-788111993.html
    https://docs.bmc.com/docs/discovery/113/tw_run_upgrade-788110385.html
     
     
      
      
        

     


    Article Number:

    000086301


    Article Type:

    Solutions to a Product Problem



      Looking for additional information?    Search BMC Support  or  Browse Knowledge Articles