8 Replies Latest reply on Aug 13, 2019 6:54 AM by Hitesh Jha

    db_verify process stuck

    Hitesh Jha
      Share This:

      Hello All,

      I have started the db_verify activity on the consolidator on 9th Aug.As checked now it is still not completed.Our appliance is still down .

      Please help .

       

        • 1. Re: db_verify process stuck
          Duncan Grisby

          Well, you could attach to the screen it's running in and see if it shows anything. Do

           

          ps auxfww

           

          to get a process tree. You can see which screen the db_verify is a child of and attach to that one with "screen -r <process_id>".

           

          If it has hung, that is a good sign that the database file it is checking is indeed corrupted somehow. What has led you to run db_verify in the first place?

          2 of 2 people found this helpful
          • 2. Re: db_verify process stuck
            Hitesh Jha

            Please refer the response as below.

            tideway  160470  0.0  0.0 118788  1636 ?        Ss   Aug09   0:00 SCREEN

            tideway  160471  0.0  0.0 108580  2228 pts/0    Ss+  Aug09   0:00  \_ /bin/bash

            tideway  161704  0.0  0.1 124908 97412 pts/0    T    Aug09   0:00      \_ /usr/tideway/BerkeleyDB.6.2/bin/db_verify p0000_nTaxonomyVersion_state

            tideway  161722  0.0  0.0  67564 46208 pts/0    T    Aug09   0:00      \_ /usr/tideway/BerkeleyDB.6.2/bin/db_verify p0000_nRoleKind_pidx

            tideway  162091  0.0  0.0 118516  1360 ?        Ss   Aug09   0:00 SCREEN

            tideway  162092  0.0  0.0 108452  2016 pts/3    Ss   Aug09   0:00  \_ /bin/bash

            tideway  162110  0.0  0.0 100956   664 pts/3    S+   Aug09   0:00      \_ tail -f /usr/tideway/db_verifyResultFile.txt

            tideway  162252  0.0  0.0 118516  1364 ?        Ss   Aug09   0:00 SCREEN

            tideway  162253  0.0  0.0 108456  2016 pts/5    Ss   Aug09   0:00  \_ /bin/bash

            tideway  162276  0.0  0.0 100956   668 pts/5    S+   Aug09   0:00      \_ tail -f /usr/tideway/db_verifyResultFile.txt

            tideway  162503  0.0  0.0 118652  1532 ?        Ss   Aug09   0:00 SCREEN

            tideway  162504  0.0  0.0 108456  2020 pts/6    Ss   Aug09   0:00  \_ /bin/bash

            tideway  162592  0.0  0.0 100956   664 pts/6    S+   Aug09   0:00      \_ tail -f /usr/tideway/db_verifyResultFile.txt

            tideway  181841  0.0  0.0 118712  1596 ?        Ss   03:46   0:00 SCREEN

            tideway  181842  0.0  0.0 108456  2024 pts/4    Ss   03:46   0:00  \_ /bin/bash

            tideway  181866  0.0  0.0 100956   668 pts/4    S+   03:48   0:00      \_ tail -f /usr/tideway/db_verifyResultFile.txt

            tideway  182204  0.0  0.0 118516  1372 ?        Ss   04:22   0:00 SCREEN

            tideway  182205  0.0  0.0 108456  2052 pts/9    Ss   04:22   0:00  \_ /bin/bash

            tideway  182224  0.0  0.0 100956   668 pts/9    S+   04:22   0:00      \_ tail -f /usr/tideway/db_verifyResultFile.txt

            tideway  182227  0.0  0.0 118788  1548 ?        Ss   04:22   0:00 SCREEN

            tideway  182228  0.0  0.0 108456  2024 pts/10   Ss+  04:22   0:00  \_ /bin/bash

            tideway  182985  0.0  0.0 108452  2052 pts/14   Ss   06:27   0:00  \_ /bin/bash

            tideway  183005  0.0  0.0 100956   664 pts/14   S+   06:27   0:00      \_ tail -f /usr/tideway/db_verifyResultFile.txt

            tideway  182670  0.0  0.0 118516  1376 ?        Ss   05:46   0:00 SCREEN

            tideway  182671  0.0  0.0 108456  2060 pts/8    Ss   05:46   0:00  \_ /bin/bash

            tideway  182689  0.0  0.0 100956   664 pts/8    S+   05:47   0:00      \_ tail -f /usr/tideway/db_verifyResultFile.txt

            tideway  182726  0.0  0.0 118516  1372 ?        Ss   05:47   0:00 SCREEN

            tideway  182727  0.0  0.0 108456  2028 pts/12   Ss   05:47   0:00  \_ /bin/bash

            tideway  182921  0.0  0.0 118372  1176 pts/12   S+   06:25   0:00      \_ screen -r 181841.pts-2.cltdvladmc01

            tideway  182960  0.0  0.0 118516  1372 ?        Ss   06:26   0:00 SCREEN

            tideway  182961  0.0  0.0 108456  2028 pts/13   Ss   06:26   0:00  \_ /bin/bash

            tideway  182981  0.0  0.0 118372  1176 pts/13   S+   06:27   0:00      \_ screen -r 182227.pts-7.cltdvladmc01

             

            After  executing the command screen -r with PID .

             

            [tideway@cltdvladmc01 ~]$ screen -r 160470

            There is a screen on:

                    160470.pts-4.cltdvladmc01       (Attached)

            There is no screen to be resumed matching 160470.

            [tideway@cltdvladmc01 ~]$ screen -r 161704

            There is no screen to be resumed matching 161704.

            [tideway@cltdvladmc01 ~]$ screen -r 160471

            There is no screen to be resumed matching 160471.

            [tideway@cltdvladmc01 ~]$ screen -r 161722

            There is no screen to be resumed matching 161722.

            [tideway@cltdvladmc01 ~]$ screen -r 162091

            BDB5105 Verification of p0006_rBMC_ApplicationSystemServices_state succeeded.

            p0006_rBMC_Component_pidx

            BDB5105 Verification of p0006_rBMC_Component_pidx succeeded.

            p0006_rBMC_Component_rels

            BDB5105 Verification of p0006_rBMC_Component_rels succeeded.

            p0006_rBMC_Component_state

            BDB5105 Verification of p0006_rBMC_Component_state succeeded.

            p0006_rBMC_Dependency_pidx

            BDB5105 Verification of p0006_rBMC_Dependency_pidx succeeded.

            p0006_rBMC_Dependency_rels

            BDB5105 Verification of p0006_rBMC_Dependency_rels succeeded.

            p0006_rBMC_Dependency_state

            BDB5105 Verification of p0006_rBMC_Dependency_state succeeded.

            p0006_rBMC_ElementLocation_pidx

            BDB5105 Verification of p0006_rBMC_ElementLocation_pidx succeeded.

            p0006_rBMC_ElementLocation_rels

            BDB5105 Verification of p0006_rBMC_ElementLocation_rels succeeded.

            p0006_rBMC_ElementLocation_state

            BDB5105 Verification of p0006_rBMC_ElementLocation_state succeeded.

            p0006_rBMC_HostedAccessPoint_pidx

            BDB5105 Verification of p0006_rBMC_HostedAccessPoint_pidx succeeded.

            p0006_rBMC_HostedAccessPoint_rels

            BDB5105 Verification of p0006_rBMC_HostedAccessPoint_rels succeeded.

            p0006_rBMC_HostedAccessPoint_state

            BDB5105 Verification of p0006_rBMC_HostedAccessPoint_state succeeded.

            p0006_rBMC_HostedSystemComponents_pidx

            BDB5105 Verification of p0006_rBMC_HostedSystemComponents_pidx succeeded.

            p0006_rBMC_HostedSystemComponents_rels

            BDB5105 Verification of p0006_rBMC_HostedSystemComponents_rels succeeded.

            p0006_rBMC_HostedSystemComponents_state

            BDB5105 Verification of p0006_rBMC_HostedSystemComponents_state succeeded.

            p0006_rBMC_InIPSubnet_pidx

            BDB5105 Verification of p0006_rBMC_InIPSubnet_pidx succeeded.

            p0006_rBMC_InIPSubnet_rels

            BDB5105 Verification of p0006_rBMC_InIPSubnet_rels succeeded.

            p0006_rBMC_InIPSubnet_state

            BDB5105 Verification of p0006_rBMC_InIPSubnet_state succeeded.

            p0006_rSyncMaintainer_pidx

            BDB5105 Verification of p0006_rSyncMaintainer_pidx succeeded.

            p0006_rSyncMaintainer_rels

            BDB5105 Verification of p0006_rSyncMaintainer_rels succeeded.

            p0006_rSyncMaintainer_state

            BDB5105 Verification of p0006_rSyncMaintainer_state succeeded.

            • 4. Re: db_verify process stuck
              Duncan Grisby

              Something is very confused here. Why have you got so many screens all doing a tail of the same file?  And screens attached to screens?  There is also something peculiar about the fact that there are two db_verify commands running, when normally we would expect there to be just one.

               

              The screen that is actually running the verify is 160470, but it thinks it's attached somewhere. Run

               

              screen -d -r 160470.pts-4.cltdvladmc01

              1 of 1 people found this helpful
              • 5. Re: db_verify process stuck
                Hitesh Jha

                Thanks Duncan.Please refer the o/p as below.

                [tideway@cltdvladmc01 datadir]$ /usr/tideway/BerkeleyDB.*/bin/db_recover -v > /usr/tideway/db_verifyResultFile.txt 2>&1

                [tideway@cltdvladmc01 datadir]$ for db in p*; do echo $db; /usr/tideway/BerkeleyDB.*/bin/db_verify $db; done  >> /usr/tideway/db_verifyResultFile.txt 2>&1 ;

                 

                 

                 

                 

                [tideway@cltdvladmc01 datadir]$

                [tideway@cltdvladmc01 datadir]$

                [tideway@cltdvladmc01 datadir]$

                • 6. Re: db_verify process stuck
                  Hitesh Jha

                  I can see 160470 is attached.

                  I failed to understand why it's taking so much long duration to verify the data store.

                  .

                   

                  • 7. Re: db_verify process stuck
                    Duncan Grisby

                    Everything is in a confused state. Reboot the appliance and start again. Don't create endless screen sessions. Don't run the verify command more than once.

                    1 of 1 people found this helpful
                    • 8. Re: db_verify process stuck
                      Hitesh Jha

                      Thanks Duncan