Share:|

 

Logging Table

 

Problem Area

Logging
Location
ITSM/SRMCombined API/SQL/Filter logging while reproducing the problem, arerror log, Bundle log$AR_SYSTEM_HOME/db
Workflow DesignerBundle log, F12 Browser Developer Mode$AR_SYSTEM_HOME/db
UIBundle log, Jetty log, F12 Browser Developer mode & Fiddler$AR_SYSTEM_HOME/db
ConnectorsBundle log, arerror log, F12 Browser Developer mode$AR_SYSTEM_HOME/db
ReportingBundle log, arerror log$AR_SYSTEM_HOME/db
MyIT IntegrationBundle log, UX debug on the MyIT side, F12 Browser Developer mode & Fiddler
$AR_SYSTEM_HOME/db
ApprovalsCombined API/SQL/Filter logging while reproducing the problem, remoteaction.log, bundle.log, arerror log

$AR_SYSTEM_HOME/db

$AR_SYSTEM_HOME/Logs

Installationarsystem_install_log.txt, startup logs

$AR_SYSTEM_HOME/Logs

/tmp

Post Install scriptsarerror log, post install log

$AR_SYSTEM_HOME/db

$AR_SYSTEM_HOME/Logs

RSSOarjavaplugin & bundle log$AR_SYSTEM_HOME/db
User Group Syncsb user synch log, combined api & SQL logging

$AR_SYSTEM_HOME/db

"/src" directory

 

Service Broker File System

 

Most of the Service Broker related files are contained within the ARSystem directory, some of the more important directories and files are highlighted in the following illustration

SB FileSystem.PNG

 

OSGI Console

 

The Service Broker application is run through a variety of different bundles which are loaded once the "arcontroller" script is run as part of an AR System restart. Unlike MyIT, we cannot restart the individual components of Service Broker (without negative consequence) however we can check Service Broker's bundle status by using the Eclipse OSGI Console (Open Services Gateway Initiative).

 

The logging level for OSGI is controlled in the  "$AR_SYSTEM_HOME/conf/config.properties" file

 

# Enable osgi console for debugging

osgi.console.enable.builtin=true

osgi.console = 8181    #Note this port has changed to 12666 for MyIT Service Broker 3.3

To connect to the console you simply telnet to the Service Broker host and execute the "SS" command.

 

[root@clm-aus-01]# telnet localhost 12666

Osgi>ss

Some of the more important Service Broker related bundles can be identified by "myservice"

 

OSGI Bundles.PNG

If you notice any of these bundles are in a "RESOLVED" state you can attempt to restart as follows:

 

Start “ID#”

In the event the Service Broker login page becomes unavailable for all users, restart the com.bmc.myservice.uiservice-war_0.0.1 and try again. If you are still unable to log in please raise a case with BMC Technical Support. Type "disconnect" to end the session.

 

These connectors and bundles are held within the $AR_SYSTEM_HOME/deploy directory.

SBDeploy.PNG

Service Broker Scripts

 

The "rxscripts" directory and sub-directories contain many scripts which are called by Service Broker but in some cases you may also run them manually.

 

/bin/check_migrations.sh        Checks migrations <SBHost>:8008/api/sbe/migrations/

/bin/create_schema.sh             Get’s called by the post install script

/bin/current-user.sh                  Checks the current user who’s in the current session

/bin/tenant/find.sh                     Lists all the tenants

/bin/tenant/delete.sh                 Deletes a tenant

/bin/connector/refresh.sh         Refreshes the available connectors

/bin/healthcheck_core.sh         Checks if Service Broker is up or down

/bin/login.sh                               Used to log into Remedy from the command line

/bin/run_migrations.sh             Get's called when DB migrations are required

/bin/setenv.sh                            Sets up the environmental variables to allow the API endpoints

/bin/tenant_seed.sh                  Get’s called by the post install script

/bin/version.sh                          Get's current version by running ..<SBHost>:8008/api/sbe/version

/bin/service/search.sh              Lists all catalog services visible by the current MyIT user

In order to run these scripts you need to set up your environmental variables by sourcing the "setenv.sh" script

[root@clm-aus-014527 bin]# source setenv.sh

 

Once you source the "setenv" script you then need to log into Service Broker from the command line and generate an authentication cookie. Once the mycookies key is generated subsequent scripts (provided they are in the same directory as the mycookie.txt) will run.

 

[root@clm-aus-014527 bin]# ./login.sh Demo password

 

eyJhbGciOiJIUzI1NiJ9.eyJleHAiOjE0NjI0NzMzNjAsIl9pbXBlcnNvbmF0ZWRVc2VyIjpudWxsLCJzdWIiOiJlSHVDUXo5ZUliUFg1UmtxaTlZUkZIMTQwbjhpWXd5SDhkc2FtdmhcL3VMZDF1dzBjbXg5UVM3Y2F2Wm85bWplR1o1T2tRMVE0MFRrQVRmY3lFdUd4Y0V6TGtVTjA0aldieVF3MDZUMGVlVkFcLzhRSEZ0ekpTVHc9PSIsIm5iZiI6MTQ2MjQ2NjA0MCwiaXNzIjoiY2xtLWF1cy0wMTQ1MjciLCJqdGkiOiJJREdBQTVWMEdFTUJDQU9HUFFFQUFBMjQ2MlgzUFciLCJfY2FjaGVJZCI6MzIxOTE2LCJpYXQiOjE0NjI0NjYxNjB9.CHebPeTu8y3JLFnIU8IjozKONMSA28xaaam9ww-e2Gg

 

[root@clm-aus-014527 bin]# ls -lrt

-rw-r--r-- 1 root root   610 Jun 15 14:59 mycookies.txt

 

In the following example I'm running the "find.sh" script to obtain information about my tenant.

 

[root@clm-aus-014527 bin]# tenant/find.sh

{"totalSize":1,"data":[{"name":"coffee","domainIdentifier":"coffee.com","tenantId":"f53893b1d382","databaseName":"ARTenantCoffee","status":"Enabled","overlayGroupId":"f53893b1d382","activationStatus":"Completed"}]}[root@clm-aus-014527 bin]#

 

Stopping & Starting Service Broker

 

We use the "arcontroller" script from the $AR_SYSTEM_HOME directory to restart Service Broker

This script also restarts ARSystem

[root@clm-aus-014527 ARSystem]#./arcontroller <stop|start|restart>

 

The "start" option calls “/sb/start.sh” which does the following:

 

  • Runs “bin/arsystem start”
  • Checks if the DB/Port is up
  • If the DB is up it calls another script which checks the bundles status
  • If bundles are up another migration script runs which checks the DB to see if anything has changed (such as JARs) and may update a column or add a new table.

 

Enabling Debug with the logback_server.xml

 

Logging level is controlled within the "logback_server.xml" which is located in the $AR_SYSTEM_HOME/conf directory

 

level=all          Best level of debug but can impact performance

level=debug   Default level of debug

 

Control the level of debug and size by editing the highlighted sections

 

Service Broker Application

        <appender name="BUNDLE"

                class="com.bmc.arsys.logging.ReconfigurableRollingFileAppender">

                <file>${com.bmc.arsys.homedir}/${com.bmc.arsys.server.dbdir}/bundle.log</file>

                <append>true</append>

                <rollingPolicy>

                        <fileNamePattern>${com.bmc.arsys.homedir}/${com.bmc.arsys.server.dbdir}/bundle.log.%i</fileNamePattern>

                        <minIndex>1</minIndex>

                        <maxIndex>10</maxIndex>

                </rollingPolicy>

                <triggeringPolicy

                        class="ch.qos.logback.core.rolling.SizeBasedTriggeringPolicy">

                        <maxFileSize>99MB</maxFileSize>

                </triggeringPolicy>

                <!-- Extra param to say log file creation at startup -->

                <needPreLoading>true</needPreLoading>

                <encoder>

                        <pattern>%d{MM-dd HH:mm:ss.SSS} %p [%c{0}][%t][%X{userName}] %m%n</pattern>

                </encoder>

        </appender>

   <logger name="com.bmc.myservice" level="all" additivity="false">

                <appender-ref ref="BUNDLE" />

 

RSSO debug can be written to the bundle log by adding the following lines under the existing bundle section.

        <logger name="com.bmc.rsso" level="debug" additivity="false">

<appender-ref ref="BUNDLE" />

        </logger>

 

ARSystem RSSO logging can be enabled by editing the $AR_SYSTEM_HOME/pluginsvr/log4j_pluginsvr.xml file and written to the arjavaplugin log within the $AR_SYSTEM_HOME/db directory.

        <!--Set root category priority to ERROR and appenders to PluginLog-->

        <root>

                <priority value ="all" />

                <appender-ref ref="PluginLog" />

        </root>

 

Jetty logging can be very useful to capture HTTP traffic and is enabled by un-commenting the following section and changing the log level

       <!-- Jetty Logging - uncomment block if jetty logging is required -->

    <appender name="JettyLog"

                class="com.bmc.arsys.logging.ReconfigurableRollingFileAppender">

                <file>${com.bmc.arsys.homedir}/${com.bmc.arsys.server.dbdir}/jetty.log

                </file>

                <append>true</append>

                <rollingPolicy class="com.bmc.arsys.logging.StoredRollingPolicy">

                        <fileNamePattern>${com.bmc.arsys.homedir}/${com.bmc.arsys.server.dbdir}/jetty.log.%i

                        </fileNamePattern>

                        <minIndex>1</minIndex>

                        <maxIndex>8</maxIndex>

                </rollingPolicy>

                <triggeringPolicy

                        class="ch.qos.logback.core.rolling.SizeBasedTriggeringPolicy">

                        <maxFileSize>128MB</maxFileSize>

                </triggeringPolicy>

                <needPreLoading>true</needPreLoading>

                <encoder>

                        <pattern>%d{EEE MMM dd HH:mm:ss.SSS yyyy} %m%nopex%n</pattern>

                </encoder>

        </appender>

        <logger name="org.eclipse.jetty.server.Server" level="all" additivity="false">

You need to perform an "arcontroller restart" for the changes to take effect

Service Broker Bundle Log

 

Almost every troubleshooting scenario will require analysis of the bundle log. Here's an example of what a "healthy" bundle log looks like.

 

05-05 15:56:26.419 DEBUG com.bmc.myservice.connectors.service.ConnectorSrvImpl testing URI http://localhost:8008/api/sbe/connect/remedy]

05-05 15:56:26.520 DEBUG com.bmc.myservice.connectors.service.ConnectorSrvImpl connector URIs [http://localhost:8008/api/sbe/connect/baopoc]

com.bmc.myservice.connectors.service.ConnectorSrvImpl connector URIs [http://localhost:8008/api/sbe/connect/ad]

 

As you can see, DEBUG is enabled and it's constantly looping around checking each connector status. The following example demonstrates how you would use the bundle log to troubleshoot a typical problem.

 

  • In the screenshot below I have submitted a Service Request in MyIT and you can see it's gone into a fault status.

 

WBYeats.png

 

 

The first step is to open a session in the “$AR_SYSTEM_HOME/db” directory and run a "tail" on the log.

  • [root@clm-aus-012457 db]# tail -f bundle.log

This command gives you a live feed of what's happening within the log.

 

  • In the example below I can see my Service Request is faulting because a required field is empty...

 

at sun.reflect.GeneratedMethodAccessor636.invoke(Unknown Source) ~[na:na]

        at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) ~[na:1.7.0_95]

        at java.lang.reflect.Method.invoke(Method.java:606) ~[na:1.7.0_95]

        at com.bmc.myservice.common.util.SrvInterceptor.invoke(SrvInterceptor.java:23) ~[com.bmc.myservice.common-1.0.00-SNAPSHOT.jar:1.0.00-SNAPSHOT]

        ... 94 common frames omitted

Caused by: com.bmc.arsys.domain.etc.ARException: ERROR (100): Required Entry ID is empty.; form: rxn:/myit-sb/ServiceRequest

        at com.bmc.arsys.server.domain.validation.EntryValidatorImpl.validateEntryId(EntryValidatorImpl.java:335) ~[na:na]

        at com.bmc.arsys.server.domain.validation.EntryValidatorImpl.validateGetEntry(EntryValidatorImpl.java:323) ~[na:na]

        at com.bmc.arsys.server.domain.validation.handler.EntryValidationHandler.validateGetEntry(EntryValidationHandler.java:79) ~[na:na]

        at com.bmc.arsys.server.domain.service.impl.EntryServiceImpl.getEntry(EntryServiceImpl.java:366) ~[na:na]

        at com.bmc.arsys.server.facade.impl.EntryFacadeImpl.getEntry_aroundBody10(EntryFacadeImpl.java:143) ~[na:na]

        at com.bmc.arsys.server.facade.impl.EntryFacadeImpl$AjcClosure11.run(EntryFacadeImpl.java:1) ~[na:na]

  • Here's another example where I submit another Service Broker Request from MyIT and the bundle log is telling me that I need to define an Work Order Manager on the ITSM side.

Caused by: java.lang.reflect.InvocationTargetException: null

        at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) ~[na:1.7.0_95]

        at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57) ~[na:1.7.0_95]

        at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) ~[na:1.7.0_95]

        at java.lang.reflect.Method.invoke(Method.java:606) ~[na:1.7.0_95]

        at com.bmc.arsys.rx.business.action.impl.ActionServiceImpl.invokeMethod(ActionServiceImpl.java:710) ~[na:na]

        ... 80 common frames omitted

Caused by: com.bmc.myservice.connect.exception.ConnectorException: ERROR (1440853): ; No work order manager group could be found. Manually select a group from the menus. If no group with an individual in the functional role of work order manager is defined, notify your System Administrator.

        at com.bmc.myservice.connect.ar.api.ArApiManager.createEntry(ArApiManager.java:448) ~[na:na]

        at com.bmc.myservice.connect.ar.api.ArApiManager.createEntry(ArApiManager.java:411) ~[na:na]

        at com.bmc.myservice.connect.ar.api.ArApiManager.createEntry(ArApiManager.java:400) ~[na:na]

        at com.bmc.myservice.connect.ar.common.services.WorkOrderService.createWorkOrder(WorkOrderService.java:14

If you are using Service Broker 3.3 you can can simply open the Request from the Console and view the failure details.

 

SBException.PNG

 

Installation and Upgrade

There is no over the top upgrade from 3.2 to 3.3. Please contact Technical Support if you are planning on migrating this data.

Please carefully follow the instructions as described in the following link

 

Performing the installation - BMC MyIT Service Broker 3.3

 

The installation script will inform you if there are any problems. Full details are available within the "arsystem_install" log which is located in the /tmp directory. To obtain a live feed on installation progress, just run a "tail" on the log as follows:

 

[root@clm-aus-014527 tmp]# tail -1000 /tmp/arsystem_install_log.txt

 

The following extract is an example of a successful installation.

 

(Dec 15 2016 03:54:59.677 PM +0530),INFO,com.bmc.install.utility.logging.Log,

  --------------------------------------------------------

  BMC Remedy Action Request System 9.5.00 install succeeded.

  --------------------------------------------------------

(Dec 15 2016 03:54:59.697 PM +0530),INFO,com.bmc.install.product.base.installer.InstallCompletionInstallationTask,

  PROGRESS EVENT {Description=[installCompletion.installCompletion.description],Progress=[100],Detail=[installCompletion.installCompletion.complete]}

(Dec 15 2016 03:54:59.917 PM +0530),CONFIG,com.bmc.install.product.base.project.runner.ProjectRunner,

  LOG EVENT {Description=[COMPLETED InstallationTask],Detail=[com.bmc.install.product.base.installer.InstallCompletionInstallationTask]}

If the installer fails and the arsystem log doesn't identify the root cause, run the uninstall script (make sure you also remove any SB DB references if present) and run the script again in verbose mode as follows:

bash -x ./install-myit-sb.sh

This command will output a detailed log to the terminal. In the event of a failure the output will stop allowing you to scroll up and identify what was happening at the time.

Please back up the Service Broker Database and file-system before installing or upgrading. In the event you encounter a problem you can quickly roll back and send the logs to BMC Technical Support.

Connector Refresh

 

There may be occurrences where your connectors are no longer visible in the Console. If this is the case try running the connector refresh script as follows:

 

[root@clm-aus-014527 rxscripts]# source bin/setenv.sh

 

./bin/login.sh hannah_admin@domain.com password

./bin/connector/refresh.sh

Approvals

 

Starting with Service Broker 3.3, the 'MyITSB- Approval' process is responsible for sending the approval request from Service Broker workflow to the ITSM Server. When a Service Request is raised from MyIT the corresponding workflow on the Service Broker machine is triggered which in-turn calls 'Request Approval' which communicates with the AR Server

SBApprovalStub.PNG

Please use the latest version of the ITSM Integration Patch from our EPD site. There have been some updates which are required for Approvals to function correctly.

As a first step we would enable filter/API logging on the ITSM Server before reproducing the problem and we'll be able to follow the process flow within the log file. The next step would be to open the "SB:ServiceRequestStub" form on the ITSM Server and check the “Error Message” box. In the example below it's complaining about a Work Order Manager not being present.

 

ApprovalStub.png

 

Make sure the SB:LocalApprovalConfiguration & SB:RemoteApprovalConfiguration Approval forms on the ITSM AR Servers have the correct information.

 

Once the Request has been approved by the manager, remedy workflow kicks in and attempts to send the data back to Service Broker via the "remoteaction.bat" file. This file along with the contents of the "remoteaction" directory are implemented by the Integration Server Patch.

 

 

If the Approval status isn't being sent back to Service Broker enable filter/API logging. You should be able to see the "remoteaction" command attempting to send the data back. There is also a "remoteaction" log file (enabled in the logback_remoteaction file) which will allow you to track the status transmission back to Service Broker.

 

 

Browser Tools

 

Once Developer Mode is enabled check the network tab response and look for any exceptions. The screenshot below illustrates how I've captured the root cause of a problem

 

SBMyIT.PNG

 

Ensure the Service Broker Server is reachable and the fully qualified domain name is being used !

 

FQDN.PNG

 

If you have a question related to the content in this Blog post please comment.

Seeing some other problem ? Please create a new discussion !