1 2 Previous Next 15 Replies Latest reply on Jul 25, 2018 2:52 PM by Todd Schaal

    NSH script error post BSA 8.9 upgrade

    Surendar Manoharan

      I am seeing "Error:Executing redirection process failed with errno =2(No such file or directory)" error messages in almost most of my NSH script jobs since last few days after upgrading BSA environment to 8.9 SP2. Any idea what might be wrong here?

        • 1. Re: NSH script error post BSA 8.9 upgrade
          Bill Robinson

          can you include some of your scripts and the job run logs so we can see the errors in context ?

          • 2. Re: NSH script error post BSA 8.9 upgrade
            Surendar Manoharan

            attached the NSH job run log and the associated script for review.

            • 3. Re: NSH script error post BSA 8.9 upgrade
              Bill Robinson

              it's getting to at least line 229 w/o any errors.  i'd try running this w/ a 'set -x' on the second line of the script so you can see the shell debug output and track down exactly what part throwing the errors.

              • 4. Re: NSH script error post BSA 8.9 upgrade
                Alan Weston

                I don't know if this is related or not, but this sounds similar to an issue we ran into after upgrading to 8.9:

                blcli redirection fails in 8.9

                 

                --

                Al Weston

                • 5. Re: NSH script error post BSA 8.9 upgrade
                  Bill Robinson

                  there's no redirection in his script.

                  • 6. Re: NSH script error post BSA 8.9 upgrade
                    Scott Rabinow

                    Bill,

                     

                    FYI, we are having a similar issue after we upgraded from 8.9 SP1 to 8.9 SP2.  So far, I have 2 different script objects that are having this same problem, one is a relatively simple Type 1 with only a single redirection operation, and the second is a relatively simple Type 2 that uses runscript to call a relatively complex external NSH script with layers of redirection occurring.

                     

                    After several hours of troubleshooting with the "simple" Type 1 script and removing most of our internal business logic, I found that the existence of a variable in the script named "path" was the cause of the redirection failure of the "echo" command performing redirection.

                     

                    Changing the name of the variable or commenting it out entirely corrected the problem.  One other interesting note - I duplicated the echo command and removed the redirection, and it was captured as expected by the agent and appeared intact in the job log.  So, the variable's existence is only interfering with the action to redirect the output to a file.

                     

                    The simple code below will reproduce the problem via a Type 1 NSH script object:

                     

                    platform="Linux"

                    path="nonsense"

                     

                    echo $platform >/tmp/test.txt

                    echo $platform

                     

                    With "path" active (line 2 not commented or otherwise altered), there will be a redirection error message in the job log, plus a log message "linux", and an empty file in /tmp.

                     

                    With "path" inactive, there will a valid file in /tmp and a single log message in the job log for each target.

                     

                    I'll be opening a support ticket shortly, in case this behavioral change isn't intentional.

                     

                    Note: I've spent a lot more time working on the Type 2 script, but now I'll go look for any other reserved words that might have snuck in there too!

                    • 7. Re: NSH script error post BSA 8.9 upgrade
                      Scott Rabinow

                      Oh, one more thing.  Our Type 1 script impacted here was written in 2014 and functioned correctly up until the 8.9 SP2 upgrade, so it worked in 8.2, 8.5 ad 8.9 SP1 over that period.

                      • 8. Re: NSH script error post BSA 8.9 upgrade
                        Dipak Gaigole

                        There are some predefined environment variables, which includes "PATH" environment variable.

                        On Windows operating system, the environment variables are case insensitive. Thus if we use an variable named "path", it is actually going to use the predefined "PATH" variable.

                        On Unix operating system, the environment variables are case sensitive. Thus any changes to "path" won't impact "PATH".

                         

                        Here is a simple example of using the "path" variable and a equivalent error.

                         

                        In short, if we change the predefined variables, we will see the impact whenever they gets used. So it is required to rectify the unintentional use of predefined variables so as to avoid this type of misbehavior.

                        Yes, in 8.9.02 the PATH environment variable was also getting used in the case of redirection.

                         

                        Surendar Manoharan: You have the PATH variables in your script. You can rename it and it should work correctly.

                        • 9. Re: NSH script error post BSA 8.9 upgrade
                          Scott Rabinow

                          Dipak,

                           

                          Thanks for the response.  I'm aware that the use of a "reserved word" in a script is generally a bad idea, and I preach that to my users at every opportunity.  Despite that, they still occasionally sneak through to Production.  We are also a Unix-majority shop, with all of the appservers on Linux and about 7K Unix target servers and about 3K Windows target servers, but we were still bitten by this situation despite the case sensitivity of Unix.

                           

                          I've got 2 points of heartburn over this apparent issue.

                           

                          1.  The automation has worked for over 4 years as is, so there was no reason to expect that a service pack within a version would have changed the behavior of these processes.

                          2.  This is NOT the first change implemented in 8.9 that has silently broken portions of our automation.  "Silently" means unannounced or undocumented, from my perspective.  Note:  I'll gracefully back down if someone points it out in the release notes.

                           

                          In addition to what I noted above for the Type 1 NSH script, since my last post, I've also confirmed that the Type 2 NSH script using runscript to execute an external NSH script has the exact same problem in that external script, with another example of a variable badly named "path".  That script isn't working 100% after renaming that variable, so there may be other badly named variables still to be found that are causing similar failures.

                           

                          BTW, still unexplained is why only the redirection to a file was impacted - the bare "echo" to the job log still worked.

                          • 10. Re: NSH script error post BSA 8.9 upgrade
                            Dipak Gaigole

                            1.  The automation has worked for over 4 years as is, so there was no reason to expect that a service pack within a version would have changed the behavior of these processes.

                            There is no change in the behavior, but the use of one of the system environment variable in the code, has actually pointed out an issue in the script.

                             

                            2.  This is NOT the first change implemented in 8.9 that has silently broken portions of our automation.  "Silently" means unannounced or undocumented, from my perspective.  Note:  I'll gracefully back down if someone points it out in the release notes.

                            There are many predefined/reserved environment variables which all are available for applications to use. As these are reserved environment variable, it is expected that those variables will not get used for any other purpose.

                             

                            BTW, still unexplained is why only the redirection to a file was impacted - the bare "echo" to the job log still worked.

                            As i said earlier: "In short, if we change the predefined variables, we will see the impact whenever they gets used." and PATH was getting used in case of redirection.

                            The echo is a shell builtin and thus won't need to refer the PATH variable. If PATH is modified, we will see the same behavior even in bash.

                            #################################

                            [root@Linux tmp]# cat  test.sh

                            #!/bin/sh

                            echo "pre echo worked"

                            which hostname

                            PATH=/tmp/foo

                             

                            echo

                            echo "post echo worked"

                            which hostname

                            [root@Linux tmp]# sh test.sh

                            pre echo worked

                            /bin/hostname

                             

                            post echo worked

                            test.sh: line 8: which: command not found

                            [root@Linux tmp]#

                            #################################

                            • 11. Re: NSH script error post BSA 8.9 upgrade
                              Scott Rabinow

                              Dipak,

                               

                              The problem I'm describing is with NSH scripts, so why is there an "echo" binary in <install_dir>/NSH/bin if it's a shell builtin in NSH?  By the way, I do realize your example is for a shell other than NSH...

                               

                              [appserver] /opt/bladelogic/NSH/bin: ls -l echo

                              -rwxr-xr-x 1 root root 418520 May 18 16:28 echo

                               

                              Please also note I boiled down the failing script to the smallest possible code to reproduce the issue.  The real scripts have dozens of lines of functioning code on both sides of the bad variable, and there are no issues with any other feature of the real script, besides the redirection of text to a file.  So, the PATH environment variable isn't being stomped, and all other binaries are being found as expected.

                               

                              We've worked around the issue for now by changing the variable names, but that doesn't change my opinion (yet) that this is a problem.

                              • 12. Re: NSH script error post BSA 8.9 upgrade
                                Todd Schaal

                                Not sure if this is related,  but after upgrading to 8.9 sp2 I'm noticing strange behavior with NSH.  It seems that stdout  is being redirected somewhere besides the console.  runnung ls from an NSH Here command I get nothing back:

                                 

                                l

                                lgpc01m2% ls //lgpc01m2/etc/*

                                lgpc01m2%

                                 

                                 

                                But if I try to list a directory that doesn't exist I get an error as expecteed

                                 

                                 

                                lgpc01m2% ls //lgpc01m2/blah/*

                                nsh: no matches found: //lgpc01m2/blah/*

                                lgpc01m2%

                                 

                                Also using nexec seems to work:

                                 

                                lgpc01m2% nexec lgpc01m2 ls /etc/*

                                /etc/adjtime                    /etc/environment          /etc/krb5.conf       /etc/pam_ldap.conf                 /etc/shadow-

                                /etc/aliases                      /etc/ethers               /etc/latrace.conf    /etc/passwd                        /etc/shadow.20171114

                                /etc/aliases.db                   /etc/exports              /etc/ld.so.cache     /etc/passwd-                       /etc/shells

                                /etc/anacrontab                    /etc/favicon.png          /etc/ld.so.conf      /etc/passwd.20171114               /etc/smartd.conf

                                /etc/asound.conf              /etc/filesystems          /etc/libaudit.conf   /etc/pinforc                       /etc/sos.conf

                                /etc/at.deny                     /etc/fprintd.conf         /etc/libuser.conf    /etc/pm-utils-hd-apm-restore.conf  /etc/statetab

                                /etc/autofs.conf         /etc/fstab                /etc/localtime       /etc/prelink.cache                 /etc/sudo.conf

                                /etc/autofs_ldap_auth.conf      /etc/fstab-orig           /etc/login.defs      /etc/prelink.conf                  /etc/sudoers

                                /etc/auto.master          /etc/gai.conf             /etc/logrotate.conf  /etc/printcap                      /etc/sudoers.2017-08-29

                                /etc/auto.misc                 /etc/group                /etc/lsb-release     /etc/profile                       /etc/sudoers.2017-09-19

                                /etc/auto.net                  /etc/group-               /etc/ltrace.conf     /etc/protocols                     /etc/sudoers.2017-10-13

                                /etc/auto.smb                  /etc/grub.conf            /etc/magic           /etc/quotagrpadmins                /etc/sudoers.2017-11-16

                                /etc/auto.u                    /etc/gshadow              /etc/mailcap         /etc/quotatab                      /etc/sudoers.2017-11-22

                                /etc/bashrc                    /etc/gshadow-             /etc/mail.rc         /etc/rc                            /etc/sudoers.last

                                • 13. Re: NSH script error post BSA 8.9 upgrade
                                  Scott Rabinow

                                  Just to circle back for anyone who may be reading, we opened a support ticket and went through our scenarios.

                                   

                                  This is confirmed to be new behavior in SP2 that redesigned some portions of the implementation of redirection to resolve old issues.  When redirection is occurring in an NSH script, you may be able to observe a "netredir" binary operating during that period, as that binary handles the redirection.

                                   

                                  After further very careful script review, our impacts were very restricted due to the overall design of our 2 scripts.  The majority of individual statements were being executed in backticks or the newer $(...) construct so they were operating in subshells that were NOT impacted by the PATH environment variable that was indeed being stomped by the "path" local variable.

                                   

                                  So, using reserved words as variables is no longer just a bad practice, it should be expected to cause script failures.  Problems of this nature may be masked by the details of your implementation, as they were in our case.

                                  • 14. Re: NSH script error post BSA 8.9 upgrade
                                    Bill Robinson

                                    what os version is nsh running from ?

                                     

                                    when you say you run nsh here and get nothing back do you mean there's no popup window ?

                                    1 2 Previous Next