12 Replies Latest reply on Dec 9, 2016 7:09 AM by Parag Desai

    Unable to setup git-repo on a local fileserver agent

    Steffen Kreis

      Hi,

       

      we are doing quite some change sin our architecture at the moment.

      So one thing we are looking at is moving the FileServer and Patch-Repos to an NFS share that gets mounted to all of our AppServers.

       

      We are then looking at setting up two agents per AppServer. One for the management of the AppServer (bound to its external IP) and the other one for the FileServer+Patch traffic (bound only to localhost).

      We then point the Patch Payload and the FileServer location to "blfs" which is an /etc/hosts entry on all of the AppServer to 127.0.0.1.

       

      For the setup of the two agents we are following the docs from here How to configure two RSCD agents on a single host - BMC Server Automation 8.9

       

      So also like in the docs the fs-agent always maps all its connections to a user called "blfsuser" which has read/write access to the NFS shares.

       

      So far so good.

       

      We started to relocate the Patch-Payload in the first stepo and this seems to be working fine. Every AppServer talks to the "local" Payload directory through its own local fs-rscd.

       

      I am now trying to re-locate our git repository to the same share, since this is located on the single FileServer host we are running with up until today.

       

      Unfortunately i am getting the fooliwng error when i try to specify the new "Local Repo Details" in the Infrastructure Management.

       

      "git --version: Command not found"

       

      Git is of course in place.

       

      Checking the rscd.log from the FileServer-Agent that was used here it all looks fine:

       

      12/02/16 11:18:26.800 INFO rscd -  127.0.0.1 28895 54730/54730 (MYROLE:MYUSER): nexec: /opt/bmc/bladelogic/NSH/bin/nexec blfs git --version
      12/02/16 11:18:26.805 INFO1rscd -  127.0.0.1 28895 54730/54730 (MYROLE:MYUSER): nexec: > 01010 Execute remote command: "git --version"

       

      So the agent correctly receives the the "call" from the localhost address it is bound to and shows no other error.

       

      I checked the AppServer.log of the console instance i am connected against and it shows me this:

       

      [02 Dec 2016 10:29:11,721] [Client-Connections-Thread-9] [INFO] [MYUSER:MYROLE:MYDESKTOPIP] [Client] Executing command : /opt/bmc/bladelogic/NSH/bin/nexec blfs   "git --version"

      [02 Dec 2016 10:29:11,747] [Client-Connections-Thread-9] [INFO] [MYUSER:MYROLE:MYDESKTOPIP] [Client] Started pid 26115: /opt/bmc/bladelogic/NSH/bin/nexec blfs   "git --version"

      [02 Dec 2016 10:29:11,874] [Nsh-Proxy-Thread-48] [INFO] [MYUSER:MYROLE:APPSERVERIP] [BLSSOPROXY] Connecting to blfs

      [02 Dec 2016 10:29:12,052] [WaitForProcessThread-PID-26115-88] [INFO] [MYUSER:MYROLE:MYDESKTOPIP] [Client] Process finished: 26115

      [02 Dec 2016 10:29:12,053] [Nsh-Proxy-Thread-47] [INFO] [MYUSER:MYROLE:APPSERVERIP] [BLSSOPROXY] copy data stop: Connection closed

      [02 Dec 2016 10:29:12,055] [Client-Connections-Thread-9] [ERROR] [MYUSER:MYROLE:MYDESKTOPIP] [Client] ERROR while executing cmd  "git --version"  return Code is 127

      error Stream is git --version: Command not found

       

      output stream is

      [02 Dec 2016 09:55:07,753] [Scheduled-System-Tasks-Thread-3] [INFO] [System:System:] [Memory Monitor] Total JVM (B): 1567621120,Free JVM (B): 546424904,Used JVM (B): 1021196216,VSize (B): 8274857984,RSS (B): 1609035776,Used File Descriptors: 301,Used NSH Proxy Threads: 0/50,Used Client Connections: 5/200,DB Client-Connection-Pool: 2/1/1/100/75/25,DB Job-Connection-Pool: 1/1/0/2/1/1,DB General-Connection-Pool: 3/2/1/50/37/13

       

       

      I have also registered the "blfs" server object in the console and ran an NSH script against it including the command "nexec blfs git --version" and it runs fine.

       

      Info    Fri, 02 Dec 2016 11:26:20    git version 1.9.4

      Info    Fri, 02 Dec 2016 11:26:20    Exit Code 0

       

       

      Is anybody running a similiar setup without issues or has any clue what is wrong here ?

       

      Cheers

      Steffen

        • 1. Re: Unable to setup git-repo on a local fileserver agent
          Jim Wilson

          Are you sure you are joining up the right dots here?  The time stamps of the messages don't seem to be consistent with tracking a single execution end-to-end

           

          For the error itself, the command is being invoked from one RSCD to a target called blfs

           

          I'd put that RSCD into debug to see if any more details can be seen, and I'd check what blfs resolves to from that target

           

          HTH

          • 2. Re: Unable to setup git-repo on a local fileserver agent
            Steffen Kreis

            Hi,

             

            yeah since i have tried this 100 times by now the two log snippets above are actually from two different runs.

             

            I already captured the DEBUG logs for the affected rscd agent, but from my perspective they don't seem to help.

             

            Please find the extract (from another attempt) attached.

             

            Steffen

            • 3. Re: Unable to setup git-repo on a local fileserver agent
              Steffen Kreis

              Urgh !

               

              I think i found the issue :-(

               

              I compared the DEBUG output of the "nexec blfs git --version" that runs fro the Git Configuration panel and the one from when i run it manually against the blfs target from an NSH script job.

               

              Git Panel:

              nexec: ***** New connection *****

              nexec: /opt/bmc/bladelogic/NSH/bin/nexec blfs git --version

              nexec: chdir ("/") = 0

              nexec: remote execution is interactive: TRUE

              nexec: uname () = 1

              nexec: setenv TERM >xterm<

              nexec: Agent version is 8.9.00.136

              nexec: Set to have all nexec I/O in UTF8

              nexec: > 01010 Execute remote command: "git --version"

              nexec: remote command: git --version

              nexec: X11 forwarding is ON

              nexec: rscdchildpid = 32573

               

              NSH Script Job:

              nexec: ***** New connection *****

              nexec: /opt/bmc/bladelogic/NSH/bin/nexec blfs git --version

              nexec: chdir ("/") = 0

              nexec: remote execution is interactive: TRUE

              nexec: uname () = 1

              nexec: setenv TERM >xterm<

              nexec: Agent version is 8.9.00.136

              nexec: Set to have all nexec I/O in UTF8

              nexec: > 01010 Execute remote command: git --version

              nexec: remote command: git --version

              nexec: X11 forwarding is ON

              nexec: rscdchildpid = 32573

               

               

              For whatever reason there are "double-quotes" around the actual command that get triggered by the Git Panel.

              When i put the exact same quotes around the command in my NSH script i directly get:

               

              Error    Fri, 02 Dec 2016 13:03:34    git --version: Command not found

              Info    Fri, 02 Dec 2016 13:03:35    Exit Code 127

               

              Which explains it all.

              I wonder how i was ever able to set this up against a stand-alone agent on our FileServer and why it behaves different for the local one ?!

               

              Steffen

              • 4. Re: Unable to setup git-repo on a local fileserver agent
                Steffen Kreis

                It gets weirder ....

                 

                Since we are using NSH proxies here i can use a local (on my desktop) NSH session and speak against blfs.

                I was surprised that this works since from my workstation blfs is not resolvable since it is just an /etc/hosts entry pointing to localhost on each of the AppServers.

                 

                But okay this works due to the proxy usage.

                 

                So i can talk to the locally listening only agent on the appserver via the proxy and here are the results.

                 

                nexec blfs git --version

                git version 1.9.4

                 

                nexec blfs "git --version"

                git --version: Command not found

                 

                Running the same against the "management" agent on the AppServer directly (the one that is listening on the external IP) i get this.

                 

                nexec blapp git --version

                git version 1.9.4

                 

                 

                nexec blapp "git --version"

                git version 1.9.4

                 

                Both of these agents are version 8.9.00.136

                What the heck ?!?!?!

                • 5. Re: Unable to setup git-repo on a local fileserver agent
                  Bill Robinson

                  can you put both rscd in debug mode (agentctl toggle) and re-run ?

                  • 6. Re: Unable to setup git-repo on a local fileserver agent
                    Steffen Kreis

                    I am getting closer !

                     

                    The issue is related to the user switch to blfsuser.

                    When we do not map to a different user other than root this works fine and we can run the nexec command with our without doublequotes.

                     

                    Whenerver we map to a different user, which is the blfsuser in our case for the FileServer agent the nexec run with double-quotes around the actual command fails.

                     

                    Right now i suppose this is related to how the agent switches to a different user and passing over the command string.

                     

                    And the fun part is that this only seems to affect the Linux agent.

                    As our Git Repo right now sits on our Solaris based FileServer which also maps all connections to blfsuser and on this platform it all works.

                     

                    I guess i should open a ticket on this.

                     

                    Steffen

                    • 7. Re: Unable to setup git-repo on a local fileserver agent
                      Bill Robinson

                      That’s interesting.  i’m assuming it happens w/ any command like ‘ls –la’ ?

                      • 8. Re: Unable to setup git-repo on a local fileserver agent
                        Steffen Kreis

                        Here is more evidence:

                         

                        Linux with mapping to blfuser:

                        [BLPROD] @ / % nexec blfs id -a

                        uid=UID(blfsuser) gid=GID(blfsuser) groups=GID(blfsuser) context=system_u:system_r:initrc_t:s0

                        [BLPROD] @ / % echo $?

                        0

                        [BLPROD] @ / % nexec blfs "id -a"

                        [BLPROD] @ / % echo $?

                        1

                         

                        Linux with mapping to root:

                        [BLPROD] @ / % nexec blfs id -a

                        uid=0(root) gid=0(root) groups=0(root),100(users) context=system_u:system_r:initrc_t:s0

                        [BLPROD] @ / % echo $?

                        0

                        [BLPROD] @ / % nexec blfs "id -a"

                        uid=0(root) gid=0(root) groups=0(root),100(users) context=system_u:system_r:initrc_t:s0

                        [BLPROD] @ / % echo $?

                        0

                        [BLPROD] @ / %

                         

                        Solaris with mapping to blfsuser:

                        [BLPROD] @ / % nexec SOLARIS_FILESERVER id -a

                        uid=UID(blfsuser) gid=1(other) groups=1(other),GID(blfsuser)

                        [BLPROD] @ / % echo $?

                        0

                        [BLPROD] @ / % nexec SOLARIS_FILESERVER "id -a"

                        uid=UID(blfsuser) gid=1(other) groups=1(other),GID(blfsuser)

                        [BLPROD] @ / % echo $?

                        0

                        [BLPROD] @ / %

                        • 9. Re: Unable to setup git-repo on a local fileserver agent
                          Steffen Kreis

                          If i am right here,

                           

                          nobody would have ever been able to setup a git repo on a Linux based FileServer with a mapping other than root !

                          Or do i miss something here ?

                           

                          Steffen

                          • 12. Re: Unable to setup git-repo on a local fileserver agent
                            Parag Desai

                            Hi Bill,

                             

                            I am able to reproduce this on one (8.9) setup out of 3. But unable to find a pattern. Initially thought to be an issue with Git Repo on RHEL 6.8 but that too is not true as one of my Git Repo on RHEL 6.8 is working fine. Any thoughts on what else could be checked here?

                             

                            Thanks,

                            Parag