Skip navigation

TrueSight Orchestration

2 Posts authored by: Carl Wilson Moderator
Share:|

I recently decided to automate some daily checks for a client that a Remedy Team would normally perform against the ARS Servers such as checking memory for the Mid Tier processes (Java Memory Allocation).  One of the factors driving this is a memory leak in Tomcat that was affecting the systems where Tomcat needed to be restarted periodically before it became unresponsive.

Due to the type of client, additional security measures are utilised to gain access to the servers in the form of secondary level access.

 

A 2 step login process is used:

 

  • First Level - Password for the "Remedy" user that changes daily (obtained via a script using SSH)
  • Second Level - Unix second level access consisting of a Username / Password combination for authorised users where commands can be executed in the Unix shell

 

Here is where I encountered some "fun" results and unexpected/inconsistent behaviour. 

 

For obtaining the First Level Password, the Unix script that ran required that you entered your Second Level Access credentials and then simulated keystrokes to pass the various screens until you could obtain the Daily Password.

What the user saw on screen vs what was actually happening in the background with the script were 2 different things.

The Unix script running this process effectively displayed everything "nicely" to the user, but the raw SSH that was returned via the SSH Adapter showed additional information that was not displayed to the user.

This ended up being a little trickier than first thought, but through trial and error managed to obtain the information required to produce the commands and prompts required.

 

Playing around with the "Prompts" was the fun bit here as this was a script that did not actually return to a system prompt and any additional keystrokes would terminate the SSH session before you obtained the output.

 

The request looked something like this:

 

<request-data>

  <ssh-request>

    <prompts>

      <prompt name="cmd">: </prompt>

      <prompt name="Username">Username: </prompt>

      <prompt name="Password">Password: </prompt>

      <prompt name="Query">press 'M' for a manual date query</prompt>

      <prompt name="Output">===========================================================================

</prompt>

      <prompt name="Newline">\n</prompt>

    </prompts>

    <targets>

      <target name="">

        <host>{hostname.com}</host>

        <port>22</port>

        <userName>qpass</userName>

        <password>ask4help</password>

        <timeout-secs>60</timeout-secs>

        <allow-unknown-hosts>true</allow-unknown-hosts>

        <use-shell-mode />

        <prompt>: </prompt>

        <establish-connection-timeout-secs>60</establish-connection-timeout-secs>

      </target>

    </targets>

    <commands>

      <command ignore-exit-code="true" prompt="Password"><![CDATA[{username}]]></command>

      <command ignore-exit-code="true" prompt="Query">

        <EncryptedData xmlns="http://www.w3.org/2001/04/xmlenc#" Type="http://www.w3.org/2001/04/xmlenc#Content">

          <CipherData>

            <CipherValue>{encrypted password}</CipherValue>

          </CipherData>

        </EncryptedData>

      </command>

      <command ignore-exit-code="true" prompt="Password"><![CDATA[\n]]></command>

      <command ignore-exit-code="true" prompt="Newline"><![CDATA[\n]]></command>

    </commands>

  </ssh-request>

</request-data>

 

The above used the OOB "SSH" process under the "AutoPilot-AD-Utilities Module".  Values for the Secondary Access are taken from the Module Configuration, thus the "<CipherData>" tags and encrypted password in the request (I have removed other sensitive information).

 

The issues started when I attempted to then log into the ARS servers (Solaris) to perform the Java Memory Check using the same logic (and prompts/commands) above, after all the prompts for the Username and Password looked the same in a SSH session so they should work exactly the same for the next session?  Wrong, this is where the assumption "that it all works the same" was my downfall.

 

Although everything appeared normal, same prompts presented for Username / Password in the Putty Sessions, it however did not work the same in the background and through the SSH Adapter.  No matter what combination I tried I could not get past the "Password: " prompt and kept receiving a error that did not make sense (as I had not encountered this before nor could I reproduce) from the system and the session was terminated.  The logs did not shed much light on what was happening other than showing the error I was seeing being thrown.

It took about half a day of investigation to reproduce the steps that were causing the behaviour and the error, which was only present when entering an invalid Username / Password combination e.g. password / password.  Entering a valid Username and wrong Password did not cause the error it just prompted for the password again.

It took a bit more digging to understand that if I sent a "enter/enter" combination I received the same error and the session was terminated.  ** A clue to what was going on **

 

Now I had the culprit, the system was sending an "Return/Enter" type command and the session was being terminated before the actual defined commands were executed.

This led down the path of "what will cause an Return/Enter command to be sent before the actual command I had listed in the request after the prompt"?

There is one setting that causes this behaviour ..... <verify-os>

 

As explained in the following discussion, the "<verify-os>" tag when excluded from the request call can introduce different behaviour across devices when using SSH:

 

How to Use Orchestrator (AO) to execute commands on networking devices (e.g. Cisco)

 

So, now I had the culprit and the potential solution, what is next?

To verify I quickly whipped up a static adapter request (XML Context Item containing the full adapter request to execute) to test some different scenarios by using the call adapter process directly.

This allowed me to narrow down the combination that worked with these Solaris servers and the login script.

 

To make this reusable once identified, I copied the OOB SSH process (and renamed) where I adjusted the XSLT to include the required element:

 

"<verify-os>false</verify-os>"

 

in the "<targets>" node.

 

I did this for all XSLT Transforms in the process.  I can then use this particular SSH process where required.

 

The XSLT looks like the following:

 

<xsl:stylesheet xmlns:xsl="http://www.w3.org/1999/XSL/Transform" version="1.0">

  <xsl:output indent="no" cdata-section-elements="command " />

  <xsl:template match="/">

    <request-data>

      <ssh-request>

        ${prompts}

        <xsl:if test="string-length('${target}')!=0">

          <!-- Dynamic target choice -->

          <targets>

            <target name="${target}" />

          </targets>

        </xsl:if>

        <xsl:if test="string-length('${host name}')!=0">

          <!-- Dynamic target specification -->

          <targets>

            <target name="">

              <verify-os>

                <xsl:text disable-output-escaping="no">false</xsl:text>

              </verify-os>

              <host>

                <xsl:text disable-output-escaping="no">${host name}</xsl:text>

              </host>

              <xsl:if test="string-length('${port}')=0">

                <port>

                  <xsl:text>22</xsl:text>

                </port>

              </xsl:if>

              <xsl:if test="string-length('${port}')!=0">

                <port>

                  <xsl:text>${port}</xsl:text>

                </port>

              </xsl:if>

              <userName>${user name}</userName>

              <xsl:if test="string-length('${private key file}')!=0">

                <private-key-file>

                  <xsl:text disable-output-escaping="no">${private key file}</xsl:text>

                </private-key-file>

              </xsl:if>

              <xsl:if test="string-length('${private key data}')!=0">

                <private-key-data>

                  <xsl:text disable-output-escaping="no">${private key data}</xsl:text>

                </private-key-data>

              </xsl:if>

              <xsl:if test="string-length('${pass phrase}')!=0">

                <pass-phrase>

                  <xsl:choose>

                    <xsl:when test="'${pass phrase encryption type}'='base64'">

                      <xsl:attribute name="encryption-type">

                        <xsl:text disable-output-escaping="no">${pass phrase encryption type}</xsl:text>

                      </xsl:attribute>

                      <xsl:text disable-output-escaping="no">$[pass phrase]</xsl:text>

                    </xsl:when>

                    <xsl:when test="'${pass phrase encryption type}'='plain'">

                      <xsl:attribute name="encryption-type">

                        <xsl:text disable-output-escaping="no">${pass phrase encryption type}</xsl:text>

                      </xsl:attribute>

                      <xsl:text disable-output-escaping="no">$[pass phrase]</xsl:text>

                    </xsl:when>

                    <xsl:otherwise>${pass phrase}</xsl:otherwise>

                  </xsl:choose>

                </pass-phrase>

              </xsl:if>

              <password>

                <xsl:choose>

                  <xsl:when test="'${password encryption type}'='base64'">

                    <xsl:attribute name="encryption-type">

                      <xsl:text disable-output-escaping="no">${password encryption type}</xsl:text>

                    </xsl:attribute>

                    <xsl:text disable-output-escaping="no">$[password]</xsl:text>

                  </xsl:when>

                  <xsl:when test="'${password encryption type}'='plain'">

                    <xsl:attribute name="encryption-type">

                      <xsl:text disable-output-escaping="no">${password encryption type}</xsl:text>

                    </xsl:attribute>

                    <xsl:text disable-output-escaping="no">$[password]</xsl:text>

                  </xsl:when>

                  <xsl:otherwise>${password}</xsl:otherwise>

                </xsl:choose>

              </password>

              <xsl:if test="string-length('${command timeout}')!=0">

                <timeout-secs>

                  <xsl:text disable-output-escaping="no">${command timeout}</xsl:text>

                </timeout-secs>

              </xsl:if>

              <xsl:if test="string-length('${connection name}')!=0">

                <connection>

                  <name>

                    <xsl:text disable-output-escaping="no">${connection name}</xsl:text>

                  </name>

                  <xsl:if test="string-length('${terminate connection}')!=0">

                    <terminate-on-exit>

                      <xsl:text disable-output-escaping="no">${terminate connection}</xsl:text>

                    </terminate-on-exit>

                  </xsl:if>

                </connection>

              </xsl:if>

              <xsl:if test="string-length('${known hosts config}')!=0">

                <known-hosts-config>

                  <xsl:text disable-output-escaping="no">${known hosts config}</xsl:text>

                </known-hosts-config>

              </xsl:if>

              <xsl:if test="string-length('${allow unknown hosts}')!=0">

                <xsl:choose>

                  <xsl:when test="'${allow unknown hosts}'='true'">

                    <allow-unknown-hosts>

                      <xsl:text>${allow unknown hosts}</xsl:text>

                    </allow-unknown-hosts>

                  </xsl:when>

                  <xsl:when test="'${allow unknown hosts}'='false'">

                    <allow-unknown-hosts>

                      <xsl:text>${allow unknown hosts}</xsl:text>

                    </allow-unknown-hosts>

                  </xsl:when>

                  <xsl:otherwise>

                    <allow-unknown-hosts>

                      <xsl:text>false</xsl:text>

                    </allow-unknown-hosts>

                  </xsl:otherwise>

                </xsl:choose>

              </xsl:if>

              <use-shell-mode>

                <xsl:text disable-output-escaping="no">$[use shell mode]</xsl:text>

              </use-shell-mode>

              <xsl:if test="string-length('${preferred pk algorithm}')!=0">

                <xsl:choose>

                  <xsl:when test="'${preferred pk algorithm}'='ssh-dss'">

                    <preferred-pk-algorithm>

                      <xsl:text>${preferred pk algorithm}</xsl:text>

                    </preferred-pk-algorithm>

                  </xsl:when>

                  <xsl:when test="'${preferred pk algorithm}'='ssh-rsa'">

                    <preferred-pk-algorithm>

                      <xsl:text>${preferred pk algorithm}</xsl:text>

                    </preferred-pk-algorithm>

                  </xsl:when>

                  <xsl:otherwise>

                    <preferred-pk-algorithm>

                      <xsl:text>ssh-rsa</xsl:text>

                    </preferred-pk-algorithm>

                  </xsl:otherwise>

                </xsl:choose>

              </xsl:if>

              <prompt>${prompt}</prompt>

              <xsl:if test="string-length('$[charSet]')&gt;0">

                <character-set>

                  <xsl:text disable-output-escaping="no">${charSet}</xsl:text>

                </character-set>

              </xsl:if>

              <xsl:if test="string-length('${establish connection timeout}')!=0">

                <establish-connection-timeout-secs>

                  <xsl:text disable-output-escaping="no">${establish connection timeout}</xsl:text>

                </establish-connection-timeout-secs>

              </xsl:if>

            </target>

          </targets>

        </xsl:if>

        <xsl:if test="string-length(.)&gt;0">

          <commands>

            <command>

              <xsl:choose>

                <xsl:when test="'${command encryption type}'='base64'">

                  <xsl:attribute name="encryption-type">

                    <xsl:text disable-output-escaping="no">${command encryption type}</xsl:text>

                  </xsl:attribute>

                  <xsl:attribute name="timeout-secs">

                    <xsl:if test="string-length('${command timeout}')=0">

                      <xsl:text disable-output-escaping="no">60</xsl:text>

                    </xsl:if>

                    <xsl:text disable-output-escaping="no">${command timeout}</xsl:text>

                  </xsl:attribute>

                </xsl:when>

                <xsl:when test="'${command encryption type}'='plain'">

                  <xsl:attribute name="encryption-type">

                    <xsl:text disable-output-escaping="no">${command encryption type}</xsl:text>

                  </xsl:attribute>

                  <xsl:attribute name="timeout-secs">

                    <xsl:if test="string-length('${command timeout}')=0">

                      <xsl:text disable-output-escaping="no">60</xsl:text>

                    </xsl:if>

                    <xsl:text disable-output-escaping="no">${command timeout}</xsl:text>

                  </xsl:attribute>

                </xsl:when>

                <xsl:otherwise>

                  <xsl:attribute name="encryption-type">

                    <xsl:text disable-output-escaping="no">plain</xsl:text>

                  </xsl:attribute>

                  <xsl:attribute name="timeout-secs">

                    <xsl:if test="string-length('${command timeout}')=0">

                      <xsl:text disable-output-escaping="no">60</xsl:text>

                    </xsl:if>

                    <xsl:text disable-output-escaping="no">${command timeout}</xsl:text>

                  </xsl:attribute>

                </xsl:otherwise>

              </xsl:choose>

              <xsl:value-of select="." disable-output-escaping="no" />

            </command>

          </commands>

        </xsl:if>

        <xsl:if test="string-length(&quot;//commands/command&quot;) &gt; 0 ">${commands}</xsl:if>

      </ssh-request>

    </request-data>

  </xsl:template>

</xsl:stylesheet>

 

This allows the generated request to include the required element in the "<targets>" section, which eliminated the behaviour I was seeing allowing the commands to execute as required.  I could then parse the output and nicely format an email to send to the Remedy Team showing the memory consumption, which I would schedule to happen daily.

 

Working request:

 

<request-data>

  <ssh-request>

    <prompts>

      <prompt name="cmd">: </prompt>

      <prompt name="Username">Username: </prompt>

      <prompt name="Password">Password: </prompt>

      <prompt name="Bash">$</prompt>

      <prompt name="bash-3.2">bash-3.2$</prompt>

    </prompts>

    <targets>

      <target name="">

        <verify-os>false</verify-os>

        <host>xxxx.arsserver.host.com</host>

        <port>22</port>

        <userName>remedy</userName>

        <password>{dailypassword}</password>

        <timeout-secs>60</timeout-secs>

        <allow-unknown-hosts>true</allow-unknown-hosts>

        <use-shell-mode />

        <prompt>Username:</prompt>

        <establish-connection-timeout-secs>60</establish-connection-timeout-secs>

      </target>

    </targets>

    <commands>

      <command verify-os="false" prompt="Password" ignore-exit-code="true"><![CDATA[{username}]]></command>

      <command verify-os="false" prompt="Bash" ignore-exit-code="true">

        <EncryptedData xmlns="http://www.w3.org/2001/04/xmlenc#" Type="http://www.w3.org/2001/04/xmlenc#Content">

          <CipherData>

            <CipherValue>{encrypted password}</CipherValue>

          </CipherData>

        </EncryptedData>

      </command>

      <command verify-os="false" prompt="Bash"><![CDATA[PID=$(/usr/ucb/ps -auxww | egrep tomcat6 | egrep -v egrep | awk '{print $2}') ; ps -p $PID -o pid,vsz | grep $PID | awk '{print $2/1024}' ; unset PID]]></command>

    </commands>

  </ssh-request>

</request-data>

 

Although this took a couple of days to diagnose, in-between all the other things I was doing, it meant I would not forget this in the future when seeing similar behaviour.

 

Some points of interest to anyone wanting to use the SSH Adapter and understand how to configure to work how they expect it to.  [This is a consolidation of various discussions available on the SSH Adapter].

 

  1. The "<prompt>" element in the "<targets>" section is required to run multiple commands in the one session without terminating after each command.  If not included the SSH Adapter will open each command in its own session e.g. 3 commands, 3 separate sessions.  If the "<prompts>" section is not used in the adapter request, then the prompt defined in the targets sections is used for all commands.
  2. The "<verify-os>" element in the "<targets>" section is required on certain types of devices to stop additional commands being issued due to the adapter attempting to verify the OS.  You will see the associated command attempting to execute with the adapter log level set to "Debug".  To eliminate, add the element into the correct adapter request section.
  3. Where you specify the "prompt" (or interchangeable with "expect" attribute) in the command to be executed, this is the prompt that you are expecting to receive once the command has executed and completed (not the prompt you are expecting before the command is run).
  4. If you have the "<prompts>" section defined, you can reference the name of the prompt defined directly in the <command>.  This adds value as you can name your prompts with something recognisable and descriptive.

 

Hopefully if you encounter similar behaviour you can use this to narrow down what the issue is and correct.

 

Enjoy and good luck process building using the SSH Adapter.

 

Carl

Carl Wilson

Useful XPath Commands

Posted by Carl Wilson Moderator Nov 30, 2014
Share:|

Introduction


As BAO is primarily an XML manipulation tool, XPath is a primary function that will be utilised above the "Basic" functions available and the following site is a great reference to start with in learning command basics:

 

http://www.w3schools.com/xpath/

 

A list of XPath functions is available here that can be used with BAO:

 

XPath, XQuery, and XSLT Function Reference

 

However, these examples list mainly the basics and there are times where more advanced XPath statements are required especially where there are attributes used to contain information and you need to extract a specific value based on an attribute.

The following list some XPath commands to take advantage of some of the more advanced hard to find syntax to extract information from an XML document.

 

XPath Examples

 

Example:

 

<Groups>

  <Group GroupType="ASSIGNEE" GroupID="MEA-CUSTID-GRP-00001">

  <Name>SV.GP.CFS.GSC BTR Service Desk</Name>

  </Group>

  <Group GroupType="RESPONSIBLE" GroupID="MEA-CUSTID-GRP-00002">

  <Name>SV.GP.CFS.GSC BTR Service Desk2</Name>

  </Group>

  <Group GroupType="ACCOUNTABLE" GroupID="MEA-CUSTID-GRP-00003">

  <Name>SV.GP.CFS.GSC BTR Service Desk3</Name>

  </Group>

</Groups>

 

Commands:

 

Extract value based on location in the XML document:

 

//Group[1]/@GroupID

<result>MEA-CUSTID-GRP-00001</result>

 

string(//Group[1])

<result>SV.GP.CFS.GSC BTR Service Desk</result>

 

//Group[2]/@GroupID

<result>MEA-CUSTID-SBSA-GRP-00002</result>

 

string(//Group[2])

<result>SV.GP.CFS.GSC BTR Service Desk2</result>

 

However the above assumes that the XML is a static document and is not changing.

If you have a "dynamic" XML document, then elements and attributes may be generated as required and therefore using a static location will not provide the intended results.  Therefore we need a method to identify and extract the information based on the attribute.

XPath commands can be embedded within each other so as to provide an extended range of functions.

The following shows one method for extracting this information.

 

Example:


<TaskIdentifiers Number="SVR17349937">

  <AltIdentifier Type="PARTNER" Number="PARTNERTSK00005"/>

  <AltIdentifier Type="MAINTASK" Number="MAINTSK00005"/>

  <AltIdentifier Type="RELATEDTASK" Number="CHM17799732"/>

</TaskIdentifiers>

 

Commands:

 

Extract information based on attribute value in the XML document (using embedded XPath):

 

//TaskIdentifiers/@Number

<result>SVR17349937</result>

 

//AltIdentifier[@Type='PARTNER']/@Number

<result>PARTNERTSK00005</result>

 

//AltIdentifier[@Type='MAINTASK']/@Number

<result>MAINTSK00005</result>


//AltIdentifier[@Type='RELATEDTASK']/@Number

<result>CHM17799732</result>



I hope this helps with demystifying XPATH which at times can be quite complex.



Filter Blog

By date:
By tag: