1 2 3 4 Previous Next 55 Replies Latest reply on May 9, 2016 9:28 AM by Brian Engel

    Error running NSH script and patch analysis on 1 machine out of 2000

    Brian Engel

      So we've had a ticket open with BMC for about a month on this issue and still can't solve it. We have 1 server out of 2000+ in our enterprise that an NSH password change job fails on. It also can't patch this machine via blade due to the inability to run patch analysis. Not sure why this one server is a problem but no one seems to know the answer.

       

      This is showing up in the application log of the server

      Faulting application name: bldeploy.exe, version: 0.0.0.0, time stamp: 0x55173ed0

      Faulting module name: ntdll.dll, version: 6.3.9600.18146, time stamp: 0x5650b9bb

      Exception code: 0xc0150004

      Fault offset: 0x00000000000ec540

      Faulting process id: 0x1378

      Faulting application start time: 0x01d18c32bbd862ea

      Faulting application path: C:\Program Files\BMC Software\BladeLogic\RSCD\sbin\bldeploy.exe

      Faulting module path: C:\Windows\SYSTEM32\ntdll.dll

      Report Id: f98d87ec-f825-11e5-8123-005056bc11fe

      Faulting package full name:

      Faulting package-relative application ID:

       

      Any help would be appreciated.

        • 1. Re: Error running NSH script and patch analysis on 1 machine out of 2000
          Bill Robinson

          what's the ticket # ?

           

          what version of bsa (appserver, target rscd) and os on the target ?

          • 2. Re: Error running NSH script and patch analysis on 1 machine out of 2000
            Brian Engel

            BMC Support Case 00091504

            BSA 8.6.01.71

            Target OS is 2012 R2

            • 3. Re: Error running NSH script and patch analysis on 1 machine out of 2000
              Jim Wilson

              I did a quick Google search for \exception Code 0xc0150004 and the top hit suggests a corrupt or missing dll

              Error Code : 0xc0150004

               

              HTH

              • 4. Re: Error running NSH script and patch analysis on 1 machine out of 2000
                Brian Engel

                Just some updates on this for you. BMC Support suspects bldeploy, so they had me run it manually. When I do I get this error:
                The program can't start because bladmin.dll is missing from your computer. Try reinstalling the program to fix this problem.


                So then when I search for bladmin.dll it's on the target server, even though the error message says it isn't.

                • 5. Re: Error running NSH script and patch analysis on 1 machine out of 2000
                  Brian Engel

                  Any chance you could check in to this Bill? We're still trying to get through this and it's moving at a snails pace. Support has been zero help and this ticket is going on well over a month. I'm lucky to hear from them once a day at the very most. Often longer.

                  • 6. Re: Error running NSH script and patch analysis on 1 machine out of 2000
                    Brian Engel

                    Deploy log shows

                    Error04/28/2016 14:31:37Application terminated unexpectedly.  This could be due to a machine reboot that was unexpectedly slow.
                    Info04/28/2016 14:31:37The bldeploy process terminated unexpectedly. See bldeploy transactions log to determine reason for failure. ( exitCode = -5005 )

                     

                     

                    Transaction Log shows:

                    04/28/16 14:31:36.214 DEBUG    bldeploy - TJM invoked with arguments:

                    04/28/16 14:31:36.217 DEBUG    bldeploy - args 0 = bltargetjobmanager

                    04/28/16 14:31:36.219 DEBUG    bldeploy - args 1 = -start

                    04/28/16 14:31:36.221 DEBUG    bldeploy - args 2 = -cmd

                    04/28/16 14:31:36.223 DEBUG    bldeploy - args 3 = bldeploy 1b40d7052e343505906101771eeb285b -N=/tmp/stage/56384546d2d03aa2bd10eb3acaa5d444 -P=<SERVERNAME> -Q=params.txt -PKG=NR24-2450842.1 -DryRun -Xr -V2 -js0 -jr0 -jc0

                    04/28/16 14:31:36.229 DEBUG    bldeploy - args 4 = -me

                    04/28/16 14:31:36.230 DEBUG    bldeploy - args 5 = 1b40d7052e343505906101771eeb285b

                    04/28/16 14:31:36.231 DEBUG    bldeploy - args 6 = -mp

                    04/28/16 14:31:36.232 DEBUG    bldeploy - args 7 = -sp

                    04/28/16 14:31:36.233 DEBUG    bldeploy - args 8 = -ps

                    04/28/16 14:31:36.233 DEBUG    bldeploy - args 9 = -h60

                    04/28/16 14:31:36.234 DEBUG    bldeploy - BlTargetJobManager started

                    04/28/16 14:31:36.235 DEBUG    bldeploy - Commandline: bltargetjobmanager -start -cmd bldeploy 1b40d7052e343505906101771eeb285b -N=/tmp/stage/56384546d2d03aa2bd10eb3acaa5d444 -P=<SERVERNAME> -Q=params.txt -PKG=NR24-2450842.1 -DryRun -Xr -V2 -js0 -jr0 -jc0  -me 1b40d7052e343505906101771eeb285b -mp -sp -ps -h60

                    04/28/16 14:31:36.244 DEBUG    bldeploy - Found path to cmd at C:\Program Files\BMC Software\BladeLogic\RSCD\sbin/bldeploy.exe

                    04/28/16 14:31:36.245 DEBUG    bldeploy - Executing "C:\Program Files\BMC Software\BladeLogic\RSCD\sbin\bldeploy.exe" 1b40d7052e343505906101771eeb285b -N=/tmp/stage/56384546d2d03aa2bd10eb3acaa5d444 -P=<SERVERNAME> -Q=params.txt -PKG=NR24-2450842.1 -DryRun -Xr -V2 -js0 -jr0 -jc0  -StartNew

                    04/28/16 14:31:36.249 DEBUG    bldeploy - ProcessExecutor::createProcess batchFile=, cmd="C:\Program Files\BMC Software\BladeLogic\RSCD\sbin\bldeploy.exe" 1b40d7052e343505906101771eeb285b -N=/tmp/stage/56384546d2d03aa2bd10eb3acaa5d444 -P=<SERVERNAME> -Q=params.txt -PKG=NR24-2450842.1 -DryRun -Xr -V2 -js0 -jr0 -jc0  -StartNew, cmdDir=

                    04/28/16 14:31:36.360 INFO     bldeploy -  <SERVERNAME> 4472 BladeLogicRSCD (Not_available): (Not_available): Getting processes details

                    04/28/16 14:31:36.869 DEBUG    bldeploy -  <SERVERNAME> 4472 BladeLogicRSCD (Not_available): (Not_available): Process Name = bldeploy.exe

                    04/28/16 14:31:36.870 DEBUG    bldeploy -  <SERVERNAME> 4472 BladeLogicRSCD (Not_available): (Not_available):   Command = bldeploy.exe

                    04/28/16 14:31:36.872 DEBUG    bldeploy -  <SERVERNAME> 4472 BladeLogicRSCD (Not_available): (Not_available):   Process ID (pid) = 3936

                    04/28/16 14:31:36.873 DEBUG    bldeploy -  <SERVERNAME> 4472 BladeLogicRSCD (Not_available): (Not_available):   Parrent Process ID (ppid) = 2488

                    04/28/16 14:31:36.875 DEBUG    bldeploy -  <SERVERNAME> 4472 BladeLogicRSCD (Not_available): (Not_available):   Process Owner Username = BladeLogicRSCD

                    04/28/16 14:31:36.877 DEBUG    bldeploy -  <SERVERNAME> 4472 BladeLogicRSCD (Not_available): (Not_available):   User ID = 0

                    04/28/16 14:31:36.878 DEBUG    bldeploy -  <SERVERNAME> 4472 BladeLogicRSCD (Not_available): (Not_available):   Group ID = 0

                    04/28/16 14:31:36.880 DEBUG    bldeploy -  <SERVERNAME> 4472 BladeLogicRSCD (Not_available): (Not_available):   Real Memory = 1796 KB

                    04/28/16 14:31:36.881 DEBUG    bldeploy -  <SERVERNAME> 4472 BladeLogicRSCD (Not_available): (Not_available):   Virtual Memory = 724 KB

                    04/28/16 14:31:36.883 DEBUG    bldeploy -  <SERVERNAME> 4472 BladeLogicRSCD (Not_available): (Not_available):   Total CPU Time = 0 seconds

                    04/28/16 14:31:36.885 DEBUG    bldeploy -  <SERVERNAME> 4472 BladeLogicRSCD (Not_available): (Not_available):   CPU Units = 0

                    04/28/16 14:31:36.886 DEBUG    bldeploy -  <SERVERNAME> 4472 BladeLogicRSCD (Not_available): (Not_available):   CPU Usage% =  0.0%

                    04/28/16 14:31:36.888 DEBUG    bldeploy -  <SERVERNAME> 4472 BladeLogicRSCD (Not_available): (Not_available):   Memory Usage% =  0.0%

                    04/28/16 14:31:36.889 DEBUG    bldeploy -  <SERVERNAME> 4472 BladeLogicRSCD (Not_available): (Not_available):   Hostname = <SERVERNAME>

                    04/28/16 14:31:36.891 DEBUG    bldeploy -  <SERVERNAME> 4472 BladeLogicRSCD (Not_available): (Not_available):   Priority = 0

                    04/28/16 14:31:36.893 DEBUG    bldeploy -  <SERVERNAME> 4472 BladeLogicRSCD (Not_available): (Not_available):   Process Start Time = Thu Apr 28 14:31:36 2016

                     

                     

                    04/28/16 14:31:36.895 DEBUG    bldeploy -  <SERVERNAME> 4472 BladeLogicRSCD (Not_available): (Not_available):   Process Group ID = 0

                    04/28/16 14:31:36.896 INFO     bldeploy -  <SERVERNAME> 4472 BladeLogicRSCD (Not_available): (Not_available): Got processes list: 1 entries

                    04/28/16 14:31:36.898 DEBUG    bldeploy - Compare to blp_prog=bldeploy.exe

                    04/28/16 14:31:36.899 DEBUG    bldeploy - Command has started

                    04/28/16 14:31:36.901 DEBUG    bldeploy - PID not valid 3936 for bldeploy

                    04/28/16 14:31:36.905 ERROR    bldeploy - Application terminated unexpectedly.  This could be due to a machine reboot that was unexpectedly slow.

                    04/28/16 14:31:36.909 DEBUG    bldeploy - Deleted event file: C:\Program Files\BMC Software\BladeLogic\RSCD\\Transactions\events\1b40d7052e343505906101771eeb285b.out

                    04/28/16 14:31:36.914 DEBUG    bldeploy - Deleted event file: C:\Program Files\BMC Software\BladeLogic\RSCD\\Transactions\events\1b40d7052e343505906101771eeb285b.in

                    04/28/16 14:31:36.917 DEBUG    bldeploy - TJM generated Exit message: exitCode=-5005: MSG=The bldeploy process terminated unexpectedly. See bldeploy transactions log to determine reason for failure. ( exitCode = -5005 ):

                    04/28/16 14:31:36.919 DEBUG    bldeploy - Operation complete return code -5005

                    • 7. Re: Error running NSH script and patch analysis on 1 machine out of 2000
                      Bill Robinson

                      are you able to run the bldeploy manually ?  and what is being deployed here?  can other bldeploys run on this target ? or just the one package is failing ?  the above is the bldeploy or the bltargetmanager log ?

                      • 8. Re: Error running NSH script and patch analysis on 1 machine out of 2000
                        Brian Engel

                        I tried running BLDeploy manually, I got the following error
                        bldeploy error.png

                         

                         

                        The package being deployed simply changes the admin password to a random, set character length sequence. Other deploys are failing as well. We can't patch this server or anything. The above is the bltargetmanager log. Where is the bldeploy log located? I will post it.

                        • 9. Re: Error running NSH script and patch analysis on 1 machine out of 2000
                          Bill Robinson

                          bldeploy should be in RSCD/Transactions/log/bldeploy-xxx.log w/ the same uuid as the targetmanager log.  it's possible if the bldeploy is outright failing that it's not generated.

                           

                          is bladmin present on the target ?  where ?

                          • 10. Re: Error running NSH script and patch analysis on 1 machine out of 2000
                            Jim Wilson

                            As a test, please

                             

                            Copy

                            bldeploy.exe

                             

                            from

                            C:\Program Files\BMC Software\BladeLogic\RSCD\sbin

                             

                            to

                            C:\Program Files\BMC Software\BladeLogic\RSCD

                             

                            Run

                            bldeploy -ver

                             

                            Let us know the result

                            • 11. Re: Error running NSH script and patch analysis on 1 machine out of 2000
                              Brian Engel

                              Yeah there isn't any bldeploy logs in there, only bltargetmanager logs.

                               

                              There is a bladmin.dll in the RSCD folder. BLAdmin is also mapped with R/W to the local admin account on the box.

                              • 12. Re: Error running NSH script and patch analysis on 1 machine out of 2000
                                Luis Curadelli

                                Hello,

                                 

                                With Brian on webexes we have:

                                 

                                1- Reinstalled the agent on this machine.

                                2- copy the bladmin.dll from a working machine to the non-working target, registered and restarted the rscd agent service.

                                3- move the pthreadvc2.dll file as indicated by Knowledge Article.

                                4- setup debug mode for rscd agent, appserver and job, re run job and also gathered bldeploy log.

                                5- checked windows event viewer, where error about bldeploy terminating due to a slow reboot error is shown.

                                 

                                 

                                 

                                • 14. Re: Error running NSH script and patch analysis on 1 machine out of 2000
                                  Jim Wilson

                                  So, we have now reproduced the problem as originally reported and per my update on April 1, this is pointing to a corrupt DLL

                                   

                                  Per the Windows Event Log:

                                   

                                  Faulting module path: C:\Windows\SYSTEM32\ntdll.dll

                                   

                                  This is looks like a Windows OS issue specific to this server (as bldeploy works OK on the 1999 servers)

                                   

                                  Have you tried the information from the link I located?

                                  1 2 3 4 Previous Next