8 Replies Latest reply on Oct 31, 2019 10:28 AM by Scott Rabinow

    Continue script after reboot

    Szymon Zycinski
      Share This:

      Hello

       

      I have an issue with such scenario.

      1. I wrote a type 1 NSH script that does something on Linux server and checks if reboot is required after that.

      2. If reboot is required it is being performed within 5 minutes - if not, it proceeds like follow.

      3. Via nexec runs local scripts that check if everything is fine if so nothing happens, but if something fails it rolls back changes.

       

      How can I reasume script that include reboot sequence.

      I want to avoid any rc.local and so on changes - I would like to use pure bladelogic instead.

        • 1. Re: Continue script after reboot
          Bill Robinson

          at a high level:

          first i would do a 'cd //@' to cd out of the target (since a type 1 will cd into the target before starting)

          then have a loop that does an 'agentinfo' continuously so you can figure out if the server rebooted or not (because it will stop responding). 

          maybe also do an uptime check after it comes back on line or after 5 min.

          then run whatever you need to run if you need to rollback

          • 2. Re: Continue script after reboot
            Szymon Zycinski

            I tried:

            nexec $NSH_RUNCMD_HOST /sbin/shutdown -r 1 "Patching completed. Reboot required. System will reboot in next 5 minutes"

                sleep 30s

                #loop until agentinfo $? = 0

            agentinfo $NSH_RUNCMD_HOST

            alive=$?

            i=0

            while [ $alive -eq 1 ]; do

                    echo "$NSH_RUNCMD_HOST is still dead ....$i"

                    sleep 10s

                    agentinfo $NSH_RUNCMD_HOST

                    alive=$?

                    i=$((i+1))

                    if [ $i -eq 20 ]; then

                    echo Something went wrong - send status as "Unrecovered from boot"

                    exit -1

            fi

            done

            sleep 30s

            cd //$NSH_RUNCMD_HOST/

            echo "Proceed with checks!"

             

            At the moment script fails at bold line with below error:

            "//@/opt/bmc/bladelogic/NSH/tmp/application_server/scripts/job__7c0bc09e-5039-4212-96de-47699af14a4b/22e5858b-72de-4fa4-8609-26bd04c141ba.script_DBKey-SJobKeyImpl-2002325-2_CentOS_patch.nsh:cd:79: no such file or directory: //servername/"

            SSL_write (Error on socket: 'Broken pipe')

            SSL_write

            • 3. Re: Continue script after reboot
              Bill Robinson

              i don't see the cd //@/ at the top before you start the reboot.

              • 4. Re: Continue script after reboot
                Bill Robinson

                well, if you are cd'd into a system and reboot it, then the path you are into goes away. it's like being cd'd into a directory when the directory gets deleted.

                 

                so after you nexec the reboot you are going to get connection errors.

                • 5. Re: Continue script after reboot
                  Szymon Zycinski

                  I did:

                   

                  cd //@

                      nexec $NSH_RUNCMD_HOST /sbin/shutdown -r 1 "Patching completed. Reboot required. System will reboot in next 1 minute"

                      exit

                      #loop until agentinfo $? = 0

                  agentinfo $NSH_RUNCMD_HOST

                  alive=$?

                  i=0

                  while [ $alive -eq 1 ]; do

                          echo "$NSH_RUNCMD_HOST is still dead ....$i"

                          sleep 10s

                          agentinfo $NSH_RUNCMD_HOST

                          alive=$?

                          i=$((i+1))

                          if [ $i -eq 20 ]; then

                          echo Something went wrong - send status as "Unrecovered from boot"

                          curl -XPOST ........

                          exit -1

                  fi

                  done

                  sleep 30s

                  echo "Proceed with checks!"

                   

                  It did not work.

                  • 6. Re: Continue script after reboot
                    Bill Robinson

                    hmm, i would try:

                     

                    cd //@

                    disconnect $NSH_RUNCMD_HOST

                        nexec $NSH_RUNCMD_HOST /sbin/shutdown -r 1 "Patching completed. Reboot required. System will reboot in next 1 minute"

                        exit

                        #loop until agentinfo $? = 0

                    agentinfo $NSH_RUNCMD_HOST

                    alive=$?

                    i=0

                    while [ $alive -eq 1 ]; do

                            echo "$NSH_RUNCMD_HOST is still dead ....$i"

                            sleep 10s

                            agentinfo $NSH_RUNCMD_HOST

                            alive=$?

                            i=$((i+1))

                            if [ $i -eq 20 ]; then

                            echo Something went wrong - send status as "Unrecovered from boot"

                            curl -XPOST ........

                            exit -1

                    fi

                    done

                    sleep 30s

                    echo "Proceed with checks!"

                    what is the target os, and what version of bsa is this ? 

                    • 7. Re: Continue script after reboot
                      Szymon Zycinski

                      host=TESTSERVER

                      cd //$host/

                      ... #lot of stuff here

                      echo "REBOOTING IN 1 minute!"

                          cd //@/

                          nexec $host /sbin/shutdown -r 1 "Patching completed. Reboot required. System will reboot in next 1 minute"

                          sleep 90s

                          agentinfo $host

                      alive=$?

                      i=0

                      while [ $alive -eq 1 ]; do

                              echo "$host is still dead ....$i"

                              sleep 10s

                              agentinfo $host

                              alive=$?

                              i=$((i+1))

                              if [ $i -eq 20 ]; then

                              echo "Something went wrong Rabbit: $host: Fail: Unrecovered from boot"

                              continue

                      fi

                      done

                      sleep 60s

                      echo GET BACK to $host

                      cd //$host/

                       

                      And the last line cd //$host/ fails with:

                       

                      /opt/bmc/bladelogic/appserver/NSH/tmp/application_server/scripts/job__9ebc2c5a-7225-4e9e-b087-050b5e94ac0d/92954d42-2618-47eb-a37c-0c1012895001.script_DBKey-SJobKeyImpl-2002329-5_patch-v2.nsh:cd:89: no such file or directory: //TESTSERVER/

                       

                      Target is CentOS 7

                      and BSA is Version: 8.9.04.200

                      • 8. Re: Continue script after reboot
                        Scott Rabinow

                        In case you might be interested in an alternative thought, long ago we gave up trying to do it all in a single script.

                         

                        We typically implement this as a 3 job sequence.  The first job would do the required work and then issue the reboot command.  The second job waits for the agent to become available again.  The third job would perform all of the post reboot tasks.

                         

                        The second job that waits for the agent to be available again can then be used in many such sequences.  We wrote ours originally in 2009 and are still using it!

                         

                        We also have a variant where this second job issues the reboot command itself.  It takes the target server name as a parameter, runs "nexec $host reboot" and then loops until the agent goes down.  Once the agent is down, it starts a fresh loop to wait for the agent to come back up.  No fancy maneuvering, and it never cd's to the target server at all...  This variant was also written in 2009!

                         

                        Both of these scripts have worked for v7.4.6 through v8.9 SP2.

                         

                        The first variant is most commonly used on Unix servers and the second variant on Windows servers.