2 Replies Latest reply on Apr 19, 2011 12:41 PM by Jim Campbell

    Execute Script Once and NSH Proxy

    Jim Campbell

      I have a "Execute script once passing the host list as a parameter to the script" that looks lke:



      for $Target in $1


      cat > //$TARGET/c/temp/Bladelogic/foo.vbs << EOF

      VBScript stuff here


      answer=`nexec $TARGET cmd /c cscript //nologo c:\\\\temp\\\\Bladelogic\\\\foo.vbs`

      blcli_execute Server printPropertyValue $answer some_property

      blcli_storeenv answer2


      echo $Target,$answer,$answer2 >> //appserver/somefile.csv





      Outside of the need for error checks, why is this script retaining connections to the nsh proxy indefinitely?  With nsh proxies set to 200, it seems to retain exactly one nsh connection for each iteration through the loop, effectively limiting me to running this against 200 servers at most.Is there something special I need to do for it to release the connection?  It seems wrong, but would I be better off running this against each host separately (even though I think i would need a separate jvm for each blcli call?).  I'm not looking for fast performance out of this so i don't mind nexecing each server one at a time, but I want to be able to (eventually) run it against thousands of servers and finish.  As it is, I eventually run out of nsh contexts and don't get results for most of the servers.

        • 1. Execute Script Once and NSH Proxy
          Bill Robinson

          I think this is KA345034 in the kb:



          NSH Scripts Using NEXEC spawn huge number of TCP connections which hang around even though command is complete. Hitting open file limit on appserver.


          LP: BMC BladeLogic Operations Manager Suite 7.6.00
          DR: BMC BladeLogic NSH Script and Script Job 7.6.00


          Details: We are running BL on RHEL appservers. We have noticed a problem affecting type 2 NSH scripts which target a large number of servers that call nexec. We have a number of these scripts which we use for discovering things about managed servers and then setting properties accordingly. For example, we pass in a large list of target servers as $1, and then use a loop within the script to query each one, often using an nexec command. We use the results of the command to set property values for the servers. However what we have seen in a few scenarios such as this, is a massive amount of open TCP connections being spawned, seemingly by NEXEC, which are not closed down after the command exists. This is causing problems as each one uses a file descriptor on the Linux appserver and the jobs eventually crash as all file descriptors are used up. We can see this information by locating the PID of the running NSH script and then using the command lsof -p <PID>. The results are like this... (this is only an example of one result)...


          script_DB 19265 bladmin 11u IPv4 1295432990 TCP rp0900002.unix.barclays.co.uk:34833->tsmpb.wload.barclays.co.uk:ssad (ESTABLISHED)


          Same can also be seen by doing netstat -ap|grep <PID>. I have seen as many as 1017 ESTABLISHED TCP connections which observing this problem. I believe that NEXEC should cleanly shutdown the TCP connection once the command has completed, but this does not seem to be the case.


          Issue Summary: NSH Scripts Using NEXEC spawn huge number of TCP connections which hang around even though command is complete. Hitting open file limit on appserver.


          As you discovered, this has nothing to do with nexec and is solely due to the call to:
          If [ -x //${SERVER.EN_US}/usr/ios/cli/ioscli ]; then


          What happens is NSH is being launched. The above command will cause NSH to initiate a connect to an agent. This connection will not be shutdown until the NSH process exits. So if you loop through 1000 hosts, this will create 1000 connections to targets. And those 1000 connections will be active until NSH is complete. We don't automatically disconnect because that would perform horribly if we had to continuously re-connect.


          What you can do is execute the NSH disconnect command. So your script currently looks like:


          If [ -x //${SERVER.EN_US}/usr/ios/cli/ioscli ]; then
          VIOSVER=`nexec $server "/usr/ios/cli/ioscli ioslevel"`
          echo "${SERVER.EN_US},WORKLOAD,ServerSpecial-VIOS-${VIOSVER.EN_US}" >> /tmp/serverprops.csv fi


          You can change that to:
          If [ -x //${SERVER.EN_US}/usr/ios/cli/ioscli ]; then
          VIOSVER=`nexec $server "/usr/ios/cli/ioscli ioslevel"`
          echo "${SERVER.EN_US},WORKLOAD,ServerSpecial-VIOS-${VIOSVER.EN_US}" >> /tmp/serverprops.csv fi disconnect


          or you can do disconnect ${SERVER.EN_US}


          and this will solve your problem.


          If you want a clearer example of what's going on here and why we don't automatically disconnect hosts, here's a better example:


          > nsh (we start NSH)
          # cd //${SERVER.EN_US} (this initiates a connection to ${SERVER.EN_US} in NSH which stays open
          this connection will stay open so that we don't have to continuously
          re-connect to the agent) # cd //${another-server} (this creates a new connection to ${another-server}. For performance
          purposes, NSH will not disconnect from the previous host by default)


          This is exactly what's happening with you but with the "if -x [..." instead of cd.


          It's for your exact case that we have the NSH disconnect command. It's a built-in command to NSH and you can use it to explicitly disconnect from a host if you know you don't want to stay connected

          • 2. Execute Script Once and NSH Proxy
            Jim Campbell

            Perfect, thanks.