5 Replies Latest reply on May 22, 2019 1:52 PM by Bob Anderson

    Out of disk space on ADDM 11.3

    Maicon Sousa
      Share:|

      At least once a week, ADDM is not available with the following message: This device has been shut down due to insufficient available disk space.

       

      My questions

      1. Should this occur even with the / usr / tideway partition being 93% used? (/ dev / sda10 Size-40G Used-35G Avail-2.7G-93% Usage)

      2. If I resize the two partitions with the largest volume, can I solve the problem?

       

      / dev / sda3 - Mounted on /

      Size-4.8G Used-1.7G Available-2.8G Use-38%

       

      / dev / sda10 Mounted on / usr / tideway

      Size-40G Used-35G Use Of Avail-2.7G-93%

       

      Previously, the application's VM disk was 78 Gigas and was now 128G with 59 free to resize

       

       

        • 1. Re: Out of disk space on ADDM 11.3
          Bernard Stern

          From the df command output, I can see that all your data is stored in the /usr/tideway filesystem. I strongly recommend you attach a new disk to your appliance and move the database part on this new device. You will then see something like

           

          [tideway@hostname ~]$ df -h

          Filesystem                                  Size  Used Avail Use% Mounted on

          /dev/sda3                                  4.8G  1.4G  3.1G 31% /

          devtmpfs                                    3.9G    0  3.9G  0% /dev

          tmpfs                                      3.9G    0   3.9G  0% /dev/shm

          tmpfs                                      3.9G  444M  3.4G 12% /run

          tmpfs                                      3.9G    0   3.9G  0% /sys/fs/cgroup

          tmpfs                                      2.0G  3.5M  2.0G  1% /tmp

          /dev/sda5                                  4.8G  20M   4.5G  1% /reserved

          /dev/sda7                                  1.7G  157M  1.4G  10% /var

          /dev/sda6                                  1.9G  6.3M  1.8G  1% /home

          /dev/sda1                                  711M  135M  524M  21% /boot

          /dev/sda10                                  40G  7.2G   30G  20% /usr/tideway

          /dev/sda8                                  1.2G  37M   1.1G  4% /var/log

          /dev/sdb2                                   90G  13G    73G  15% /mnt/addm/db_data

          /dev/sda9                                  852M  40M   752M  5% /var/log/audit

          tmpfs                                      782M    0   782M  0% /run/user/1000

           

          Make sure to use a large enough disk to accomodate for future data growth.

          3 of 3 people found this helpful
          • 2. Re: Out of disk space on ADDM 11.3
            Maicon Sousa

            The linux team is struggling to mount the new disk in ADDM. The disk was mounted but when using the user interface to move the files error is displayed. I think the disk was set up in the wrong way.

             

            Error during disk configuration: Configuration error: invalid / etc / fstab

             

            Can you tell me the best way to proceed?

             

            new disc.PNGerror.PNG

            1 of 1 people found this helpful
            • 3. Re: Out of disk space on ADDM 11.3
              Greg Sacrey

              I have run into this a few times. There are some things I use to get it back. 

               

              First off I agree, get your data store on a different mount.  Now that said...

               

              Use caution and please ensure you are in the correct directories before removing files.

               

              First, check for "core" files in /usr/tideway/cores:

              cd /usr/tideway/cores

              ls -la

              If there are core files, see this KA:  000028149 - ADDM: you have found one or more core files in /usr/tideway/cores. What should you do before deleting them?

               

              If no core files proceed:

               

              So, the question is: What files can you remove safely (as a temporary solution) from the ADDM appliance, so that you can get the Services to start??

               

              Please be very careful with these commands.

               

              Before removing files, it is a good idea to make sure all the services are stopped:
              sudo /sbin/service tideway stop

               

              You can delete your record and pool data:

               

              cd /usr/tideway/var/record

              rm -rf *

              cd /usr/tideway/var/pool

              rm -rf *

               

              You can delete data that was gathered for support:

               

              cd /usr/tideway/var/gather

              rm *

               

              You can delete all or some of your ADDM log files (but only those in /usr/tideway/log directory!):

               

              cd /usr/tideway/log

              rm *

               

              You can also delete any core files from /usr/tideway/core, and if you take backups of ADDM, you can consider removing backups you no longer need.

               

              Now, try starting the services:

              sudo /sbin/service tideway start

               

              FYI, how to find your top ten largest directories and files in /usr/tideway:

               

              du -h /usr/tideway | sort -n -r | head -n 10

              3 of 3 people found this helpful
              • 4. Re: Out of disk space on ADDM 11.3
                Greg Sacrey

                You can also look at compressing your datastore.

                 

                To compress the datastore, you must log in to each appliance separately.

                once logged in, run the following on each server:

                 

                screen

                sudo /sbin/service tideway stop

                tw_ds_offline_compact --clean-saam-history --smallest-first --no-questions

                 

                When complete

                sudo /sbin/service tideway start

                 

                To view the last time a compaction was run:

                Open an SSH session to the appliance. login as the tideway user, and execute the commands below

                 

                grep "tw_ds_offline_compact starting" /usr/tideway/log/*compact*
                grep "Compaction complete" /usr/tideway/log/*compact*

                Example:
                [tideway@<your appliance>]$ grep "tw_ds_offline_compact starting" /usr/tideway/log/*compact*
                2019-1-28 13:18:15: tw_ds_offline_compact starting...
                2019-1-28 13:25:01: tw_ds_offline_compact starting...

                 

                [tideway@<your appliance>]$ grep "Compaction complete" /usr/tideway/log/*compact*
                2019-1-28 13:24:06: Compaction complete.
                2019-1-28 13:27:00: Compaction complete.

                 

                 

                Hope this helps.

                4 of 4 people found this helpful
                • 5. Re: Out of disk space on ADDM 11.3
                  Bob Anderson

                  You mention that previously the VM had 78G and now it hasd 128G:

                  'Previously, the application's VM disk was 78 Gigas and was now 128G with 59 free to resize'

                   

                  How was this done?  Did the VMWare team simply 'extend' the existing disk or was another disk added?

                   

                  The process to make the additional disk space available to the appliace depends on the answer to how the additional space was made available.

                   

                  If another disk was added, then just use the GUI interface to move the datastore data to the new disk.

                   

                  If the existing disk was extended, there is a completely different way to do this.

                   

                  It is not documented/supported by BMC because it involves reworking the partion tables, but I have done this partition table work many times and documented how to do this with RHEL.  I need validate that this works on11.3 on CentOS 6/7.

                   

                  The way Tideway/BMC organized the filesystem on the appliances makes expanding possible and relatively simple, but scary when doing so without proper instructions.

                   

                  I'll share when I have validated, unless someone else has already done this and can share.

                   

                  hth

                   

                  Bob