Skip navigation
1 2 3 Previous Next

Remedy AR System

111 posts
Share This:

Welcome to May’s new AR Server Blog post and we are discussing Archiving.

 

The Archiving feature available with BMC Remedy ITSM helps to maintain system hygiene and performance. It’s the feature with which you can have control over the growing data in the system.

 

The Archiving feature archives transaction data for the following BMC Remedy ITSM applications and components:

 

  • BMC Change Management
  • BMC Knowledge Management
  • BMC Service Desk
  • BMC Service Request Management Approval
  • Task
  • BMC Remedy AR System Email Messages

 

The following record types are not archived:

 

  • BMC Requester console records
  • Foundation data
  • Process setup

 

Important thing to note here is, we do not archive system forms like User, Group, Server Statistics and Server Events by using copy to archive and delete from source because these forms use reserved fields that should exist only on one form in remedy AR System.

 

Archiving Benefits

 

Archiving data regularly provides the following benefits:

 

  • Reduces the size of your production data sets
  • Improves overall system performance (for example, searches run more quickly, because they look only at production data and not at archived data)
  • Enforces organizational data-retention policies

 

Using the AR System Archive Manager Console, you can administer the archiving policies that tells the system when to move records from the production forms to the archive forms.

 

Continuous archiving employs a mechanism that automatically archives records, by default, every 24 hours. Continuously archiving records limits the strain an archive run has on the system. By limiting the strain on the system, you help to ensure performance and reliability compared to archiving a larger volume of records, for example, every three months.

 

Archive Manager console is display only form. It gets data from back end form AR System Archive Policy which is a regular form. There are workflows triggered from back end to get the required data to Archive Manager Console.

 

Example:

 

By default Archiving is not enabled on the form.

 

 

 

 

To Enable Archiving, go-to Definitions tab and enable Archive. Select the Archive Type example: Copy to Archive and Delete from Source.

Then check the checkbox ‘Include in Archive Policy’ and Save.

 

 

 

 

Once the archiving is enabled, archive policy (Test Archive) is visible in Archive Manager Console.

Then the policy can be controlled using Archive Manager Console example: Disable/Re-enable policy/Modify the qualification/Run policy on demand, etc.

 

 

 

The "Submit Date" on the archive form does not represent the Create date of a record, it will show the date when the archive record was created. This is a designed behavior.

However Original Request ID and Create Date of archived records is still there and its copied to different Field ID on archive form and they are preserved in the archive form

 

 

Archive Metadata

 

Archiving is supported for Regular and View forms only and not supported for Display only, vendor or join forms.

Since remedy 9.x Archive forms are auto generated which means the forms already exists in the system.

If by any chance, you are unable to see the archive form, can go-to form definitions tab in Developer Studio and enable the archive policy.

 

Archive Associations

 

Associations are used when we want to establish a relationship between two or more forms in BMC Remedy Developer Studio. This relationship ensures that all the related form data is archived along with the main form when we run the archive process.

 

To define associations that need to follow during form archive:

  • Open the form with which you want to archive
  • Select the Definitions tab
  • Expand the Associations to Follow for Archive panel
  • Select the option example: ‘Selected’. Then the archive qualification given for Parent form will be used for all records from associated forms for archiving, which are selected.
  • Save the form

 

Example: When association is enabled for the AR System Email Messages form:

 

 

 

Note: We cannot create associations for join forms and archive forms.

 

Multi-threaded Archiving

 

This feature is introduced in Remedy 1902.

 

  • It Increases Archive Operation Throughput
  • Used when Archiving many records
  • Multiple threads simultaneously Archive each form
  • Num-Archive-Threads is the parameter configured from Centralized Configuration
  • To configure multi-threaded archiving, go-to AR System Administration Console --> System -->General --> Centralized configuration --> Select com.bmc.arsys.server.shared and add Num-Archive-Threads
  • Default value can be configured is 4. AR service restart is not required for changes to take effect
  • However, one Form is Archived at a time.  i.e. A record from a form will be archived based on the given qualification, then its associated record and then the next record from the form will be archived

 

 

 

 

Log snapshot of Multi-threaded Archiving. The archive log is same in all 9.x versions of remedy, except since 1902 as due to multi-threaded archiving. It will show the number of worker threads for archival.

 

5.4.png

 

 

Reference Information for additional details

 

Communities:

How to configure archiving for AR server 9.X

AR System Archive Manager is really really slow

Operating Mode

 

Documents:

 

Multi-Threaded Archiving:

 

Videos:

Share This:
Share This:

If we look at the last 5 years while we continued to deliver new features, we also changed the technology stack for Remedy, pivoted towards resilient upgrade, simplified deployment of custom code as well as completely changed the way changes made to applications are offered to customers. The attached deck will walk you through highlights of the value that an upgrade to latest version of the Remedy platform will bring in. The intent is to engage administrators in reviewing values that have been brought in and be able to create a case for routine upgrades, thus always being LIVE on the latest version of the platform. This will allow you to get a quick glance into: ease of monitoring, efficient integrations, single pane of glass for configurations for a Remedy Admin, benefits for applications by enabling archive, almost real time search, changes to the model which came in through refactoring primarily for performance gains and visibility into the environment.

Share This:

In my earlier blog I talked about the archive feature as a way to bring in hygiene, maintenance; eventually leading to better end user response times. The platform has granular controls at several layers when it comes to permissions and what fields one could see, whilst the row level security feature controls ticket data access. Permissions defined on request ID in combination with the assignee group(s) allow these controls to be enforced.

The 19.02 release of the Remedy Action Request system brought in the subquery algorithm targeted to provide performance benefits for user facing operations. Design time support was made available in 19.08. Take a read at .

Improve performance by using RLS algorithms - Documentation for BMC Remedy Action Request System 19.08 - BMC Documentati…

Look through the attached performance numbers, especially the ones which show up for Smart Reports. As an example an Asset Details report which used to take 4 mins [0.2 million records result set] now takes 21 seconds OR in case of a end user trying to get results off WorkOrder:SLMDetails [result set 0.1 million] now shows 2x improvement.

Combined with the auto discovery feature which is being made available as a hot fix in prior releases, the platform will now pick for you which are the forms on which the subquery algorithm will give you efficiencies. The way the discovery happens is by comparing execution time against a configurable interval, which is 5 seconds by default. This will allow recommendations come up for your database, for your use cases; not needing you to look at heuristics or needing you to extrapolate for these use cases.

Share This:

Deploying D2P Package to upgrade Application or to apply Server Hotfix in a Large Server Group environment can be a time-consuming process.

 

From Remedy version 19.02 onwards, you can apply the Deployment Package Simultaneously on all Servers to speed up the upgrade process. This will help to reduce the time each server in a Server Group would need to deploy a package.

 

Warning: You should be careful when using this method as the entire server group might go down when arpayloaddeploymentutil.bat

is executed and AR Servers are restarting. In such a case it’s a good practice to not include Administrator Server as part of simultaneously deployment or rollback.

 

 

Steps to deploy a package Simultaneously across servers in a Server Group

 

  1. Log in to Mid Tier as an Administrator or a package deployer user by using the following URL:
    https://<midTierServerName>:<port number>/arsys
  2. From the IT Home page, select Applications > AR System Administration > AR System Deployment Management Console
  3. In the Deployment Management Console, IMPORT the package example: ITSM 19.08

    Deployement Management console.pngReady to deploy.PNG
  4. Once Package status change to 'Ready to Deploy', don't select the DEPLOY option yet, Open AR System Monitor Form and sort the Host Name based on the Monitor Type
  5. Select the Servers where you want to perform Simultaneous Deployment and update their Rank to Rank 1 or any other Rank which will be the same for all servers. Save the changes.

      AR System Monitor OOB.png

  

   6. Go back to the Deployment Management Console and select the DEPLOY option now for this package. Refresh the panel until the status change to

        'Pending Deploy'

      DeployITSm1908.png

   

    7. Select VIEW on this package entry

    8. Locate the 'Deployment Payload' item type entry and select 'View Payload Status'

      View Payload Status.png

 

    9. Here you will see all the servers from the Server Group with the same Monitor Type, for example, ARServer or  Midtier and they will have the same Rank

and all Host Status will be 'Waiting for Utility Run'. This confirms all servers are ready for deployment.

      All servers same rank.PNG

 

  10. Run the arpayloaddeploymentutil.bat on all servers now which have the Host Status 'Waiting for Utility Run'.

  • Note: You don't have to wait for Server Host Status on Server to change to 'Deployment Success' before running the arpayloaddeploymentutil.bat on another server.

                    Run the utility on all servers and remember to exclude the Administrator Server.

       arpayload bat ran.PNG

 

  11. Use the Administrator Server to login and View the Package one more time. (Step 7 & 8). You will see all the Servers are deploying the payload simultaneously.

      post bat is ran.PNG

 

  12. Navigate back to the Deployment Management Console to confirm that the packages were Deployed Successfully across all servers.

      deployed.PNG

 

 

You can Rollback a D2P package simultaneously as well. Many steps for package simultaneous rollback are similar to package simultaneous deployment. You will see those below.

 

Note: BMC doesn't recommend to rollback a Successfully deployed package. You can choose to Rollback those packages if an issue occurs during the validation of the package;
Or if an error occurs while deploying the package, you can restore the objects on your server to the pre-deployment state.

 

 

Limitation: From Remedy Version 19.02 onwards, we have D2P package rollback criteria to avoid accidental rollback.

  • You can rollback a package within 48 hours of successful package deployment.
  • This interval can increase up to 168 hours. You can increase this only before package deployment.
  • This can be configured using ‘D2P-Rollback-Timeout-Interval’ setting which can be added in the Centralized Configuration under component com.bmc.arsys.server.shared’.

           rollback cri.png

 

Steps to rollback a package Simultaneously across servers in a Server Group

 

  1. Open AR System Monitor Form and search the Servers where you want to perform Simultaneous Rollback. Update their Rank to Rank 1 or any other Rank which will be the same for all servers (See Step.5).
    Save the changes. Remember to exclude the Administrator Server.
  2. Navigate to Deployment Management Console and select 'Rollback' for a package. Refresh the panel until the Status Change to 'Pending Rollback'

        package rollback.PNG

Note: Post 48 hours we will get an error “Rollback Timeline Validation Failed: Rollback is not allowed 48 hours after successful deployment” if this parameter is not set as the default value is 48 hours. For more details, review:  https://communities.bmc.com/docs/DOC-117790

        rollback error.PNG

 

 

  3. Select VIEW on this package entry and locate the 'Deployment Payload' item type entry and select 'View Payload Status'

         pending rollback.PNG

 

   4. You will see the list of Servers with the same monitor type here, all with status 'Waiting for Utility Run'.

        Run the arpayloaddeploymentutil.bat on all servers now which have the Host Status 'Waiting for Utility Run'.

      Note: You don't have to wait for Server Host Status on Server to change to 'Rollback Success' before running the arpayloaddeploymentutil.bat on another server.

        Run the utility on all servers and remember to exclude the Administrator Server.

        arpayload bat ran.PNG

 

   5. Use the Administrator Server to login and View the Package. You will see all the Servers are rolling the payload simultaneously.

       utility ruyn.PNG

 

   6. Navigate back to Deployment Management Console to confirm the packages were changed to Rollback successfully across all servers.

 

 

Reference Information for additional details

 

Communities

 

Document

 

Videos

Share This:

Updated --  April 17, 2020

 

 

 

 

Before you proceed

 

DISCLAIMER: There are inherent risks associated with implementing the configuration changes and instructions set forth in this document. Implementing the configuration changes described below may render your system vulnerable to unauthorized access by third parties.  Please consult with your IT and Security professionals for specific guidance about your operating environment before implementing any of the configuration changes or instructions suggested in this document.  This document and the instructions contained in it are provided “AS-IS,” and BMC disclaims all warranties and liabilities arising from your implementation of the configuration changes and instructions in this document. BMC is not responsible for any network issues arising from devices and components not provided by BMC.

 

Challenge

In the wake of recent stay-at-home orders by governments around the world due to the COVID-19 pandemic, BMC has received several inquiries from customers seeking to connect their remote employees to BMC’s Remedy IT Service Management (ITSM) offering.  The increased traffic on customers’ VPNs has been excessive and is believed to be the cause of disruptions in employees’ abilities to reach the Remedy ITSM service.

 

To enable remote employees to continue accessing Remedy ITSM to enter tickets, check the status of their tickets, and resolve tickets, several customers have asked BMC to provide instructions for an internet-facing portal into Remedy that does not require the use of VPN.

 

This blog post describes a possible approach to configuring your settings in a manner that allows you to access Remedy ITSM without going through a VPN. However, we recommend that you consult your security and IT teams before carrying out these instructions.

 

Solution

Locate the Remedy web tier (Mid Tier) in the corporate DMZ to allow external users to get to the Remedy web components while keeping the overall Remedy solution secured.

A screenshot of a mapDescription automatically generated

Resource: Remedy security architecture and controls

 

Suggested project approach

 

  • Form a cross-functional team Include everyone: security, business users, IT, and so on.
  • Follow change control procedures Ensure that all changes are implemented with the knowledge and consent of everyone impacted in your organization.
  • Perform changes on a development or test environment first Work through issues before making the changes in production. We can’t stress this enough!
  • Unit test with a group of testers that have different job roles and, if needed, different geographiesThe external testing of the Remedy Mid Tier will be very similar to the testing you would use to certify an upgrade. Utilize the same test procedures and scripts for this use case.
  • “Good internet access is required for testers to minimize or eliminate internet timeout issuesA home internet user usually has enough bandwidth and speed to connect to Remedy.

 

    • However, if you have a user with slow network connections, it might be because the user is physically far from the web server.
      • In this case, you can do the following:
      • Create simplified views of the needed forms.
      • Increase the number of seconds for the timeout.

Cautions and recommendations

 

Use the latest version of Remedy

 

Remedy version 20.02 or later is recommended.

Apply the latest patches and fixes to your Remedy deployment on a continuous basis

Reference: Downloading the installation files (select your version of Remedy)

Make sure Remedy administrators use a VPN to access the system

Administrators should not be allowed to access Remedy from non-VPN connections.

Enforce strong password complexity, a password expiration policy, and an account lockout policy

Reference: Access control, passwords and server security

Enforce HTTPS and enable TLS v1.2 only. Use of CA-signed certificates is recommended.

Reference: Security planning

Tasks

Set up a web server in your corporate DMZ Work with your internal security team to harden your server in the DMZ with typical security settings, such as disabling all non-essential ports, applying the latest patches, changing all default passwords, and so on.

Reference: https://www.cisecurity.org/cis-benchmarks/ for benchmarks.

 

See the FAQ section for common questions and answers for the security team.

 

Create and apply an SSL certificate to the new test web server for secure web sessions Ask your security team to help if needed.

Reference: Configuring the Mid Tier web server for SSL certificate

 

Configure Single Sign-On Most customers have Single Sign-On (SSO) solutions like Okta. Follow the same configuration you use for your current Mid Tier servers.

Reference: Remedy Single Sign-On Deploy the new test Mid Tier server into the DMZ If it’s not there already, deploy the new test Mid Tier server into your corporate DMZ. Test the new web server in the DMZ Test in the following order:

 

Navigate to the default web login page.

 

Success: The Remedy Mid Tier server is accessible through the internet without a firewall.

Attempt to log in with your user name and password.

Success: The Remedy Mid Tier server can connect to a Single Sign-On solution such as Okta and to the internal Remedy server.
  • Failure authenticating – Check the authentication server connection
  • Failure reaching the Remedy server – Check the firewall settings and port settings

Have users test the deployment by running through typical test scenarios such as creating tickets, checking status, working on tickets, running queries, looking at dashboards, running reports, and so on. Reference: Performing upgrade testing

Success: The test system works for these scenarios.

Check performance. Reference: Available Resources to Ensure your Environment is Configured for Peak Performance

Success: The system successfully handles your remote workforce. Employees might still experience issues with their connection to Remedy ITSM. Should this occur, you may want to ask such employees to upgrade to a business-class service with their internet service providers.

Change the configuration of the Mid Tier server to point to production You are now ready to point the Mid Tier server to the production Remedy AR System server.

 

Reference: Remedy Mid Tier configuration After you complete the configuration changes, do a quick sanity test of the key functionalities to make sure everything is working as expected.

 

Frequently Asked Questions (FAQ)

Can I test everything internally with a “standard” server first before placing the web server into the DMZ?

Yes. One method we see customers using is to create the test server internally and get everything up and running internally. When everything works as expected, including testing, the Security team then locks down the Mid Tier server (especially port numbers) and puts the Mid Tier server into the DMZ. If anything stops working as expected, the most likely root cause is a security setting, or a required port is disabled.  This method can make troubleshooting much easier because everyone knows the test server worked internally before moving into the DMZ.

 

Is Encryption on by default with Remedy?

Yes. Only encrypted clients can communicate with the Mid Tier server, which is the default setting.

Reference: Activating encryption

 

What ports need to be open to allow a Mid Tier server to communicate from the DMZ to the internal network? In order to harden security for the DMZ, the Security team will want to shut down all non-essential ports, especially any ports that typically have unencrypted traffic. Typically, an on-premises deployment of Remedy uses a portmapper to dynamically assign port numbers to some processes. When communicating from the DMZ to the internal Remedy server behind a firewall, you need to specify the port number for each Remedy component. Do not use the portmapper.

 

References

 

What are the recommended Remedy configuration settings for Mid Tier users?

 

In Remedy version 9.1.02 or later, the following parameters default to false (F). You do not need to change the configuration settings.

 

Allow-Guest-Users

 

F — Do not allow guest users. Unregistered users have no access to the system.

Allow-Unqual-Queries

F — Do not allow unqualified searches.

In Remedy version 9.0 or earlier, consider changing these parameters to false (F) in Centralized configuration. Notes:

  • Set Allow-Unqual-Queries – F after you successfully deploy and user test. Unqualified queries, especially against very large tables, can bring back very large results. This might be okay on an internal-only, high-speed local area network, but it can be slow over an internet connection.
  • You might have existing user queries, reports, or dashboards that are using unqualified queries, such as Query all HelpDesk tickets, Query all Change Management Tickets, and so on. Explore what might be impacted and test the setting.

 

A single Mid Tier server is having performance issues because of very high traffic volume. Can I add additional Mid Tier servers?

Yes. Remedy Mid Tier is designed to be put into a server pool to distribute high volumes of traffic among multiple servers Set up additional Mid Tier servers in a very similar manner to the first Mid Tier server.

Reference: Configuring the Mid Tier connection pool

 

Additional BMC Documentation resources

Additional BMC Communities resources

 

 

Have a question, make a comment below

 

For broader picture, see  COVID-19 Response Resources

Share This:

Welcome to March’s new AR Server Blog post and we are discussing D2P Log Analysis.

 

When working with the Deployment Manager Console, if you experience an issue, the following logs are automatically generated that can be used to troubleshoot the problem:

      • ard2pplugin.log: Collects Plug-in related logging. Always resides on AR Server
      • ard2pclient.log: Collects logs only when you are deploying a package using the command-line interface
      • arfiledeployer.log: Collects logs specific to the binary payload in a package
      • ard2pdeploymentactivity.log: Collects Filter, SQL, and API combined logging. Always resides on AR Server
      • armonitor.log: Collects logs for AR, Mid Tier, and Smart IT application activities

 

 

The default location of all these logs:

ComponentOperating System
Location
AR System ServerWindows<Install directory>\BMC Software\ARSystem\Arserver\Db
Linux<Install directory>/BMC Software/ARSystem/db
Mid tierWindows<Install directory>\BMC Software\ARSystem\midtier\filedeployer\Logs
Linux<Install directory>/bmc/ARSystem/midtier/filedeployer/Logs
Smart ITWindows<Install Directory\Smart_IT\filedeployer\Logs
Linux<Install Directory/Smart_IT/filedeployer/Logs>

 

 

Now let's see the different symptoms of package deployment and logging details

 

 

Stage
SymptomsLogging DetailsNext Steps
Package Import Fails
  • Unable to view the package after importing the ZIP file
  • Package import fails with 'Init Error'
  • The package fails to upload the ZIP file, and the status gets stuck in the importing state
  • Package import fails with the following error message: Error in plugin: ARSYS.ARF.ARMIGRATE

Import Failure Example 1:

  • Open ard2pplugin.log and locate the first line of the package import, search for "*/ARMigratePlugin starting import operation". Review the Date & Time on the above line to find the current attempt
    • You will find the import operation with package zip name, E.g. */ARMigratePlugin starting import operation.<Package Entry ID: E.g. ITSM2002.zip".
  • Review a couple of lines from this step to locate more details on what the import operation did and if the import was successful, E.g. ARException trying up update status to application!  form=RDA:DeploymentDataDetails status=1entryId=ITSM2002.zip.zip message=Error: Results from import log file:
  • As the message=Error, search for "ERROR -" from above line to locate the complete error message

import error.PNG

 

Import Failure Example 2:

import error 2.PNG

 

Import Success Example:

  • If package import is successful, you can search for "ARMigratePlugin: Import of Package completed" to get the complete details on what all files got imported. Review the Date & Time on the above line to find the current attempt
    • E.g. */Success: ARMigratePlugin: Import of Package completed.  Please check Log results that follow for Specific results:

import success.PNG

Review Troubleshooting package import issues
Package Deploy Fails

Package deploy fails with following error on AR System Deployment Management Console

  • Init Error
  • Init Pending Deploy

     

     

Deploy Failure Example 1:

  • Open ard2pplugin.log and locate the first line of the package deploy, search for "*/ARMigratePlugin : starting deploy". Review the Date & Time on the above line to find the current attempt
  • E.g. "/ARMigratePlugin: starting deploy init operation. <Package Entry ID: <id> Package Name: E.g. ITSM 2002, Package Version: <version>
      • Deploy operation will start only when the Import Operation is completed successfully. You will find "./ARMigratePlugin: import operation completed" message right before starting deploy
  • The first stage of deploy operation is "Deploy Init". If the package status on Deployment Management Console shows "Init Error", then in the log, search for "*/ARMigratePlugin: starting deploy init operation"
      • Review a couple of lines from this step to locate more details on the error message

deploy error 1.PNG

 

Deploy Failure Example 2:

  • The second stage of deploy operation is "Deploy Package". If the package status on Deployment Management Console shows "Init Pending Deploy", then in the ard2pplugin.log log, search for "*/ARMigratePlugin: starting deploy package item operation"
      • Review a couple of lines from this step to locate more details on the error message. Review the Date & Time on the above line to find the current attempt.
  • If the next couple of line shows "*/ARMigratePlugin: deploy package item operation completed" without any error, then open arfiledeployer.log to get further details on failure

deploy success 1.PNG

 

For package deploy, open arfiledeployer.log

  1. Search for "Connected to ARServer <hostname>". Review the Date & Time on the above line to find the current attempt.
    • Once AR Server is connected, review next few lines to confirm if the registration with AR System Monitor GUID is successful and CCS Settings are loaded
      • E.g. "Registering monitor in AR System Monitor with existing GUID <AGGAA5V0FWSPCAP03LG9PJ6T1ZOBIV>" (Monitor ID of this AR Server from AR System Monitor Form)
      • E.g. "Loading CCS settings for component ARServer_<hostname>_7319_AGGAA5V0FWSPCAP03LG9PJ6T1ZOBIV" (Hostname, Port and Monitor ID from AR System Monitor Form)
    • Search for "com.bmc.arsys.filedeployer.PayloadProcessor"
        • E.g. com.bmc.arsys.filedeployer.PayloadProcessor  - Found 3 payloads which are waiting for utility run."

     

    • If log show entry with "PayloadProcessor" and points to the correct number of payloads, it means package deploy operation is successful and you should continue with deployment payload operations.

    pending deplo success.PNG

     

    • If log stops writing post "Loading CCS settings for component", it means the package deploy operation has failed.

    pending deplo1.PNG

    Review Troubleshooting package deploy issues

    Package Deployment Payload Fails
    • The status of Deployment Payload entries is stuck in Waiting For Utility Run
    • The status of Deployment Payload entries in the package is any of the following:
      • Rollback Success
      • Download Failed
      • Deployment Failed
      • Rollback Failed
      • Skipped
      • Timed Out
      • Monitor Unreachable

    Deployment Payload Failure Example 1:

    • Locate the 'Deployment Payload' item entry from the D2P package and select 'View Payload Status' to confirm if payloads got deployed successfully or not
    • Copy the Deployment ID of the entry which shows 'Rollback' or any other status except success

    rollback.jpg

     

    • Open arfiledeployer.log and search using the GUID and find the first line, E.g. "Processing payload with DEPLOYMENT ID IDGAA5V0G75KCAPRSYOGPQVPGIG8TZ". Review the Date & Time on the above line to find the current attempt
      • Review a couple of lines from this step to locate more details and search for "payload as DEPLOYMENT FAILED"  (For any status other than success, the payload will always use deployment failed keyword)
      • Locate the line exactly above "payload as DEPLOYMENT FAILED" to see the actual cause of failure

      deployment payload error 1.PNG

       

      Deployment Payload Failure Example 2:deployment payload error 2.PNG

       

      Deployment Payload Failure Example 3:

      • Locate the 'Deployment Payload' item entry from the D2P package and select 'View Payload Status' to confirm if the host status is stuck in "Waiting For Utility Run" post running the arpayloaddeploymentutil.bat
      • If so, search for "Failed to read the PayLoad form", there is a possibility AR Server is not getting signal post utility run

      deployment payload Error 90.PNG

       

      • In such a case, open armonitor.log, search for 'Process Failed to Start'. Match the time from arfiledeployer.log and confirm if any AR Server process is stuck or failed to start

      armonitor.PNG

       

      Deployment Payload Success Example:

      deployment success.PNG

       

      Review Troubleshooting deployment payload issues

       

       

      Reference Information for additional details

       

      Share This:

      This is the fourth and final in a series of topics that will cover securing an entire Remedy ITSM system. This will be presented in several blog posts each meant to be read and reviewed in serial to each other.

       

      These sets of topics are meant to help guide a Remedy System Administrator into first understanding the potential threats to a properly configured Remedy ITSM environment. The goal is to introduce certain scenarios and facilitate conversation and policy creation, then eventually start you towards action items.  While this guide will attempt to cover many topics, your specific environment might have different business needs.  For these extra business needs, such as Government oversight or contractual requirements, you will need to consult in conjunction with a security specialist that both understands the Remedy environment as well the needs for your business.  While we will attempt to cover a large scope of items, including items outside of BMC’s control, it is inevitably up to the Remedy Administrator, System Administrator, Network Administrator, and the organization to understand the environment, plan the environment, and configure the environment for the implementation of a secure Remedy ITSM system.

       

      This will start with an overview of common industry standard security concepts.  Then we’ll review the questions that you should ask yourself and be able to answer to know what steps you need to take and to what degree do you need to implement these security requirements.  Also, we’ll review the specific Remedy products, and finally we’ll review the individual actions you can complete to improve the security of your environment.

       

      At this point in this series, you should have a beginners understanding of concepts of securing your environment, a grasp on the requirements your company puts forth to you, and have built some initial requirements and documentations on what you have planned on implementing.  This topic is going to focus on the actual implementations of some of the common items we've been discussing and building towards.  This will work as a collection of a lot of the common actions you'll need to perform from other knowledge sources.  This can also act as a type of checklist of some of those common items.  You can also use this as a discussion to tell others about items you have done or provide easier, or better, instructions that you followed.

       

      Operational Implementations

       

      Document your assets!  (Use Remedy Asset Management!)

      1. Servers (including hardware components)
      2. Other hardware, like network switches, firewalls (and even the layer the firewall is at), or load balancers.
      3. Operating system version (as well as Service Pack and patch level)
      4. Software installed on systems (Remedy products, Oracle Java, Apache Tomcat, Database, and others!)
      5. What components are installed on each system individually as well as how they might work together and communicate.
      6. Document any configuration changes to the system.  (Registry, AR System, User account, firewall rules, etc.)
      7. Any other critical data points including: who has access to the system, SSL Certs assigned to that system (including exp date), location, and other.

       

      Proceed to create and implement your own operational and maintenance policies

      1. Upgrades, patches, and hotfixes.
      2. System Monitoring Practices
      3. Regular backup policies.
      4. Regression/penetration testing.
      5. Standard change policies.
      6. Disaster Recovery policies.

       

      Use Centralized Configuration to manage Remedy, and keep track of configurations (change control)

      1. Configure Mid-Tier to sync with Centralized Configuration
        1. https://docs.bmc.com/docs/ars1902/centralized-configuration-for-the-mid-tier-836454209.html
        2. This allows for easier management of AR System Mid-Tiers.
      2. Creating a backup of system settings with Centralized Configuration.
        1. https://docs.bmc.com/docs/ars1902/backing-up-and-restoring-centralized-configuration-settings-836455155.html
        2. This can be used to create a baseline that can be used to compare against future updates to ensure system integrity and protect against unauthorized changes to the system.

       

      Technical Implementations

       

      Database Server Changes

      1. Enable Database Encryption (Oracle)
        1. This should be performed by a Oracle DB Admin.
        2. https://docs.oracle.com/database/121/DBSEG/asoconfg.htm#DBSEG020
        3. Trending in Support: Encrypting Data Between AR Servers and Oracle Databases
      2. Enable Database Encryption (MS SQL)
        1. This should be performed by a SQL Admin.
        2. https://docs.microsoft.com/en-us/sql/database-engine/configure-windows/enable-encrypted-connections-to-the-database-engine?view=sql-server-2017
        3. Trending in Support: Enabling SSL Encryption for AR to MS SQL Database Connections with Remedy 9.1 SP2 and Later
      3. Set up Windows (Domain) level authentication
        1. This should be set up by a SQL Admin.
        2. https://docs.microsoft.com/en-us/sql/database-engine/configure-windows/change-server-authentication-mode?view=sql-server-2017
        3. The Remedy Administrator should then configure Remedy to utilize the new login.
        4. https://docs.bmc.com/docs/brid1902/configuring-the-microsoft-sql-server-837866164.html#ConfiguringtheMicrosoftSQLServer-…
      4. Set up Database Server Monitoring:
        1. https://docs.microsoft.com/en-us/sql/relational-databases/performance/server-performance-and-activity-monitoring?view=sql-server-2017
        2. Oracle Database 2 Day + Performance Tuning Guide, 20c
      5. Implement DISA STIG’s for Database Servers:
        1. This is traditionally done by a Database Administrator.
        2. https://www.stigviewer.com/stig/oracle_database_12c/
        3. https://stigviewer.com/stig/ms_sql_server_2016_database/
        4. https://stigviewer.com/stig/ms_sql_server_2016_instance/

       

      Operating System Changes

      1. Monitoring Servers
        1. This is traditionally done by a System Administrator.
        2. https://access.redhat.com/documentation/en-us/red_hat_enterprise_linux/6/html/deployment_guide/ch-system_monitoring_tools
        3. https://docs.microsoft.com/en-us/windows-server/administration/server-manager/server-manager
      2. Monitoring Applications
        1. This is traditionally done by a System Administrator.
        2. Windows Servers have had this built in for several generations using the Server Manager Dashboard.  https://docs.microsoft.com/en-us/windows-server/administration/server-manager/server-manager
        3. https://access.redhat.com/documentation/en-us/red_hat_enterprise_linux/6/html/deployment_guide/ch-system_monitoring_tools
      3. Running Remedy as a User, not as System/root.
        1. Even if you don’t choose to use SQL Server Windows Authentication, you should still run the Remedy process (as well as all other components) as a user instead of as the system.  This provides a additional system level of security and auditing for the system as a whole.
        2. Remedy - Server - Tightening Remedy AR System security
      4. Setting read/write permissions to correct folders for user.
        1. Prevent the “Remedy User” from having access to other applications so it can’t interfere with them.  Prevent other Users from having access to the Remedy Applications.
        2. https://docs.microsoft.com/en-us/windows/win32/fileio/file-security-and-access-rights
        3. https://access.redhat.com/documentation/en-US/Red_Hat_Enterprise_Linux/4/html/Step_by_Step_Guide/s1-navigating-ownership.html
      5. Implement DISA STIG’s for Operating Systems
        1. This is traditionally done by a System Administrator.
        2. https://stigviewer.com/stig/windows_server_2016/
        3. https://stigviewer.com/stig/red_hat_enterprise_linux_7/

       

      Java Changes

      1. Set up JMX Monitoring of all critical services.  These can be enabled either temporarily or permanently depending on company policy.  While these settings are traditionally used to monitor performance, often performance issues can be the first sign of something gone wrong.  Having this enabled and available at any time will reduce troubleshooting time and allow for more transparency on what the system is working on.
      2. AR System Server
        1. https://bmcsites.force.com/casemgmt/sc_KnowledgeArticle?sfdcid=kA014000000dx3dCAA&type=FAQ
        2. https://communities.bmc.com/docs/DOC-111047
      3. Tomcat Servers (Mid-Tier, SmartIT, and others)
        1. https://tomcat.apache.org/tomcat-8.0-doc/monitoring.html
        2. https://tomcat.apache.org/tomcat-8.5-doc/monitoring.html
        3. https://tomcat.apache.org/tomcat-7.0-doc/monitoring.html
      4. Java Plugin Servers
        1. https://communities.bmc.com/docs/DOC-105970
      5. Review and Implement items from the Oracle Java Security Guide (These configurations may not apply to OpenJDK):
        1. https://docs.oracle.com/en/java/javase/11/security/index.html
        2. https://docs.oracle.com/en/java/javase/12/security/index.html
      6. Implement DISA STIG’s for Java.
        1. https://www.stigviewer.com/stig/java_runtime_environment_jre_version_8_unix/
        2. https://www.stigviewer.com/stig/java_runtime_environment_jre_version_8_windows/

       

      Tomcat Changes

      1. Implement DISA STIG’s for Tomcat (as a web server)
        1. https://stigviewer.com/stig/web_server/

       

      Network Changes - These steps would traditionally be set up with a Network Administrator.

      1. Implement firewall rules that only allow the communication that is necessary.  A proper Remedy System Architecture Diagram will be required to adequately perform this step.
        1. https://docs.microsoft.com/en-us/windows/security/threat-protection/windows-firewall/windows-firewall-with-advanced-security
        2. https://access.redhat.com/documentation/en-us/red_hat_enterprise_linux/6/html/deployment_guide/s1-configure_the_firewall_to_allow_incoming_ntp_packets
      2. Set up network monitoring (this is dependent on your networking equipment and a 3rd party external tool will be the tool used more than likely.  Consult your network administrator on what should/could be monitored here)
      3. There also may be other configurations that need to be made.  Please consult the vendor of your networking hardware.

       

      Note: All of the above monitoring solutions are not always required, but are highly recommended.  These solutions mentioned above are either free solutions or solutions included with their respective products already.  No additional purchasing required.  There are additional products out there that allow you to monitor each of these components both from BMC and other vendors.  These third party products can offer several advantages like notifications, centralized management, reporting, and automatic correction of issues among other features.  A example of a tool like this is BMC TrueSight Operations Management. If you’re interested in learning more, speak with your BMC Sales Representative.

       

      Even if you’re not contractually held to use a DISA STIG to control your system, they are developed by rules supplied by the National Institute of Standards and Technology and can help provide a outline to securing these systems.  These rules provide a baseline of a system that would be considered “secured” and would generally be a good idea for any organization that is concerned about security.

       

      Remedy Mid-Tier Configuration

      1. Mid-Tier Configuration Remedy Mid Tier configuration - Documentation for Remedy Action Request System 19.02 - BMC Documentation
      2. Firewalls Configuring the mid tier through a firewall - Documentation for Remedy Action Request System 19.02 - BMC Documentati…
      3. Shared service Configuring Remedy Mid Tier as a shared service - Documentation for Remedy Action Request System 19.02 - BMC Doc…
      4. Setting config.properties file (This can be managed from the Centralized Configuration Form) Settings in the config.properties file - Documentation for Remedy Action Request System 19.02 - BMC Documentation
      5. Adding a SSL/TLS Certificate to Tomcat.  (Mid-Tier, Smart Reporting, RSSO, Smart IT) Configuring the Mid Tier web server for SSL certificate - Documentation for Remedy Action Request System 19.02 - BMC…
      6. Disabling WebDAV (This is disabled by default in Tomcat) https://cwiki.apache.org/confluence/display/TOMCAT/Tomcat+WebDav
      7. Set the HTTP transport method to POST
        1. Arsystem.xmlhttp.get = false
      8. Configure the CSP Header
        1. https://docs.bmc.com/docs/ars1902/configuration-settings-a-b-836455158.html#ConfigurationsettingsA-B-header

       

      Installing Remedy Premium/Performance Security

      1. Configuring Remedy Encryption Security - Documentation for Remedy Action Request System 19.02 - BMC Documentatio…

       

      Email Engine

      1. Configuring SSL/TLS for Email Engine https://docs.bmc.com/docs/display/ars1902/Configuring+SSL+for+the+email+engine
      2. Secure incoming and outgoing emails https://docs.bmc.com/docs/display/ars1902/Securing+incoming+and+outgoing+email

       

      REST API

      1. Configuring SSL/TLS for REST API (Jetty Server) Configuring the REST API by using SSL certificates - Documentation for Remedy Action Request System 20.02 - BMC Document…
      2. Configuring OAuth with REST API Enabling OAuth authentication by the REST API with Remedy Single Sign-On integrated - Documentation for Remedy Action Re…

       

      LDAP Plugin's

      1. Configuring SSL/TLS for LDAP connections Enabling LDAP plug-ins for SSL connections post-installation - Documentation for Remedy Deployment 20.02 - BMC Documenta…

       

      Reset (System) Application Server passwords

      1. https://docs.bmc.com/docs/display/ars1902/Resetting+the+Application+Service+password

       

      Configuring AR System Users to have correct row level permissions to the data.

      1. Remedy comes with several preconfigured groups and roles based ITIL best practices.  Using these in your organization can ensure that users are not able to view, submit, or modify records that they do not have access to based on support groups, applications, or other defined rules.
      2. If needed, you can customize this by creating new users, groups, and roles.  BMC does not recommend modifying OOTB Users, Groups, or Roles.  Please see Creating Users, Groups, and Roles for more information: https://docs.bmc.com/docs/display/ars1902/Creating+users%2C+groups%2C+and+roles

       

      Using Audit Records to monitor and track records history.

      1. https://communities.bmc.com/community/bmcdn/bmc_it_service_support/blog/2013/09/06/the-pulse-auditing-your-assets
      Share This:

      This is the third in a series of topics that will cover securing an entire Remedy ITSM system. This will be presented in several blog posts each meant to be read and reviewed in serial to each other.

       

      These sets of topics are meant to help guide a Remedy System Administrator into first understanding the potential threats to a properly configured Remedy ITSM environment. The goal is to introduce certain scenarios and facilitate conversation and policy creation, then eventually start you towards action items.  While this guide will attempt to cover many topics, your specific environment might have different business needs.  For these extra business needs, such as Government oversight or contractual requirements, you will need to consult in conjunction with a security specialist that both understands the Remedy environment as well the needs for your business.  While we will attempt to cover a large scope of items, including items outside of BMC’s control, it is inevitably up to the Remedy Administrator, System Administrator, Network Administrator, and the organization to understand the environment, plan the environment, and configure the environment for the implementation of a secure Remedy ITSM system.

       

      This will start with an overview of common industry standard security concepts.  Then we’ll review the questions that you should ask yourself and be able to answer to know what steps you need to take and to what degree do you need to implement these security requirements.  Also, we’ll review the specific Remedy products, and finally we’ll review the individual actions you can complete to improve the security of your environment.

       

      Up until now we've been reviewing concepts and planning out our approach to securing our environment.  All of the steps up we've discussed up to this point, could really be applied to virtually any system you'd implement in any company, but this outline is going to get more into the specific for the Remedy ITSM system itself, and how it might be different from other systems.  After reviewing this, you might find yourself going back and changing your original requirements.  This is not only a good thing, it is expected.  Your system requirements should be fluid and ever changing as your needs and understanding of the product changes.

       

      Now that we have a clear understanding of what our system will look like, what purpose it serves, and what kind of requirements we have, we’ll need to now understand the products that we’re getting ready to use, their dependencies, and the roles and responsibilities needed to properly administer the system for day to day operations and long term success.  The first thing that will need to be accounted for are updates on the Remedy products, this should have been one of the first things you came up with in 101 - Concepts and 201 - Planning during the assigning Operation and Maintenance actives.

       

      Product Updates

       

      There are 4 types of "updates" the Remedy product will have:

       

      1. Release - These are MAJOR version changes of the Remedy products.  Remedy 9 is the latest major release.
      2. Version - These are MINOR releases of the Remedy products.  The latest version at the time of writing this is 20.02.  This is read in a “YearYear.MonthMonth” cadence to help you signify the version number to the actual release date to better understand and track upgrade opportunities.  (Version 20.02 is still in the Release 9 family.)
      3. Patch – Patches are released on a regular cadence for each version.  These will include fixes both functional and security as well as some new features.  It is always recommended for you to be on the latest patch level.  The latest patch was 19.02 Patch 1 (19.08 and 20.02 has no patches as of the time of the writing), it can also be known as “19.05” as it was released in May, but it is officially referred to as 19.02 Patch 1.  At this time, 19.08 and 20.02 does not have any patches planned for the future, all fixes will be addressed in hotfixes.
      4. Hotfix – Hotfixes are released on a as needed basis for each version/patch.  These will include fixes both functional and security and will not add new features.  It is always recommended for you to be on the latest hotfix level.  New hotfixes (typically) will only be released for the most recent patch of any currently supported version.  These are updated regularly and should be checked often.

       

      But this bring us to how to check for the newest versions, patches, and hotfixes?  This is different depending on which type of update you're looking for.

       

      Releases and Version

      Check the BMC Doc’s page for Remedy Platform to see if there is a new version listed.  If found, there will be instructions for “upgrading” found in the Doc’s page for that version.

      https://docs.bmc.com/docs/lpremitsm/remedy-platform-and-remedy-it-service-management-documentation-873371643.html

      Patches

      Check the BMC Doc’s page for the Remedy Platform Version that you currently have installed under “Release Notes and Notices”.  This will include any notices for this version (both security and functional) as well as any notices for patches.  The patch will include instructions for applying the patch. https://docs.bmc.com/docs/display/ars1902/Release+notes+and+notices

      https://docs.bmc.com/docs/display/ars1902/Release+notes+and+notices

      Hotfixes

      Hotfixes can be found for all AR System products at the link below.  They will come with their own set of instructions to follow. ftp://ftp.bmc.com/pub/ARRecommendedFixes/

      ftp://ftp.bmc.com/pub/ARRecommendedFixes/

       

      This is a general outline, and is mostly true for all Remedy products including, but not limited to, AR System Server, Atrium CMDB, Atrium Integrator, ITSM Suite, Service Request Management, and Service Level Management.  Each product will have their own documentation page, downloads, and procedure for apply the versions, patches, and hotfixes.

       

      These can all be checked for manually.  Additionally, these can always be “subscribed” to by clicking the link below:

       

      https://webapps.bmc.com/prdalertsubr/index.html

       

      By doing so, you will be alerted by email any time there is a product update which includes major security vulnerabilities, patches, and new versions.

       

      It is important to always be on the latest version of a software that you can possibly support, as this will often be the only way to mitigate any vulnerabilities that are found.  Not to include other functional and performance improvements that we're able to make along the way.

       

      You can learn more about Remedy Version strings at the following Communities post which is updated regularly as new version and patches come out.  This document will not be updated and is not the place to discuss specific issues or questions with the version strings, just to educate you on the granularity so you can better understand how to maintain your system.

       

      Remedy Release Version Strings

       

      There are other dependencies that you might have in the Remedy ITSM system that will also require you to do regular maintenance task.  Here is a list of a few common dependencies and a few key highlights that have been gathered for you.  You should ALWAYS consult the vendor of each of these products with any questions in regards to licensing, usage, installing, patches, and other operations and maintenance questions you might have.  BMC can not direct you on how to manage these components for your environment, the information below is provided as is and contains no implied warranty or declaration of support.

       

      Java

       

      1. Overview -  Java SE | Oracle Technology Network | Oracle
      2. Licensing - The current version of Java - Java SE 11 is available from Oracle under an open source license at JDK Builds from Oracle . Java SE 8 remains free of charge for general purpose desktop and server use and is available under the Oracle Binary Code License (BCL) at Java SE - Downloads | Oracle Technology Network | Oraclehttps://bmcsites.force.com/casemgmt/sc_KnowledgeArticle?sfdcid=kA01O0000015FLhSAM
      3. Oracle Roadmap - Oracle Java SE Support Roadmap  Java has recently changed their schedule for releases as well as supportability.  It is VERY important that you understand this so you can ensure that you’re not vulnerable for certain attacks.  Oracle has a open source option, LTS options, non-LTS options, feature based releases (GA) and security based releases (BPR).  You should contact Oracle to better understand all of these factors in your decision for which release, version, and your upgrade/installation path and plan.
      4. Remedy must have a defined Java Instance.  This requires the java path to be absolute.  Best practice is to put Java in a “common” or “generic” folder such as D:/Oracle/Java/JRE or etc/Oracle/Java.  This allows you to manage the path more easily, as the Oracle Installer naturally puts Java in a specific path that matches its version number.  This change to the Java installation makes the upgrade process much less painful, allowing you to more quickly perform regular updates.
      5. Where to go - Java SE - Downloads | Oracle Technology Network | Oracle

       

      Tomcat

       

      1. Overview - Apache Tomcat® - Welcome!
      2. Tomcat is a open source Java Servlet.  It is developed and available on the Apache License version 2.  Licenses
      3. You can learn more about the technology and roadmaps here: Apache Tomcat® - Which Version Do I Want?
      4. The Remedy Mid-Tier comes with a installation of Tomcat for your use in a development or non-production system.  BMC does not recommend using the bundled Tomcat installation in a production or secure environment.  You should install Tomcat separately to the Mid-Tier component and should be installed before you do the BMC Mid-Tier, Smart IT, Digital Workplace, or Remedy Single Sign-On products.  There is a documentation portal for the version you plan on using on the Apache Tomcat homepage: Apache Tomcat® - Welcome!
      5. Just like the Java installation, installing Tomcat to a “common” or “generic” folder location will allow for easier upgrades of the Tomcat application for more quickly and regular updates.

       

      SQL/DB Server

       

      This should always be handled by a SQL Administrator.  Oracle DB and Microsoft SQL Server are their own intimate parts and are constantly changing.  Please consult the BMC documentation to determine product compatibility and chose your version based on this information.  Contact Microsoft and Oracle for more information on these products and how to support them.  BMC Product Availability and Compatibility

       

      Operating System

       

      This should always be handled by a System Administrator.  Red Hat/IBM Linux and Microsoft Windows Server are their own intimate parts and are constantly changing.  Please consult the BMC documentation to determine product compatibility and chose your version based on this information.  Contact Microsoft and IBM/Red Hat for more information on these products and how to support them.  BMC Product Availability and Compatibility

       

      Keeping all of the software components above up to date is the primary way to fight against software defects that result in vulnerabilities.  As time goes on, vulnerabilities are discovered and corrected.  This happens at the hardware, system, and software levels.  Having a plan in place to ensure you're running the latest software version possible should be a high priority.

       

      Understanding Remedy ITSM

       

      The next item that we need to understand is the Remedy AR System Architecture itself.  This can be very overwhelming at first, but is something that you need to break down one item at a time.  The below diagram will outline a network diagram of a single server ITSM system and all the processes that it would have, your environment WILL vary.  Each component is detailed as a separate "server" in the diagram, but one single hardware server can host many of these processes (like in a development environment) or they can be spaced out infinity (like in a high availability production environment).

       

      AR System Technological Diagram.png

      Note: The above diagram will be replaced with a interactive version in the future.  For now, it might be best to download this image locally, and view in a application that will allow you to zoom in and out, as there are several components that have text next to them that will be helpful.

       

      Remedy ITSM Connections

       

      Here are some key takeaways from the diagram above:

       

      1. Everything starts with AR System Server.  This should be the first component installed, upgraded, patched, tested, and monitored in any Remedy ITSM environment.  All connections either start with AR System Server or start with the goal of eventually reaching AR System Server and is the center of a ITSM system.
      2. All the plugins start up with Remedy AR System Monitor, which is configurable by the armonitor.cfg file.  In a typical environment, only Email and Flashboard plugins are started separate of AR System Server (Windows).
      3. Each server has its own configuration and logs.  While they also share some configurations.  This is defined on a per server basis and is not a common theme that you can always rely on.  The diagram above includes the name of the process that is running (what you can see in task manager) along with any relevant configuration files and the default port setting or the setting itself you can change to manage the port numbers.  All settings should be managed using the Centralized Configuration form if possible.  Centralized configuration - Documentation for Remedy Action Request System 20.02 - BMC Documentation
      4. The communication between each plugin server and AR Server is critical.  And this communication must work for the whole system to work. Firewall rules may need to be implemented to allow this communication.  Even if you don’t use some of these components, like FTS, this communication should still be available to the server it can be used by other components.  All of these ports are configurable.
      5. Client components (like Mid-Tier, Developer tools, and more) are all started up separately of AR System Server and will tend to error out UNLESS AR System Server is up and running and is available for communication.  This is why it is important to ensure AR System Server is set up and working first.  These are also configurable ports.  But, it must match the settings configured for AR System Server.
      6. This diagram is to be used as a outline for you to build your own architecture.  It does not consider things like a load balancer, custom plugins, email servers, and other items.  This diagram was built with the intention to mimic a single server environment for a full ITSM System, so your environment will more than likely be even more complex and might have many other requirements.

       

      External Connections

       

      External connections often go out onto the network and might cross several other systems before they end at their destination.  This includes LDAP servers used for authentication, third party web services, and even database integrations like Atrium Integrator connections.  But this also will include connections like Mid-Tier or Smart IT to AR System Server when they’re on different systems (recommended configuration for a production system).  These will often require not just local configuration, but also configuration of additional network devices like routers and firewall appliances.  A Network Admin and System Admin should be engaged for all activities dealing with network connections.

       

      Internal Connections

       

      Internal connections, almost always, stay locally and “loopback” to the same server. Usually, these will include things like the plugin servers.  Almost always these connections only require a local configuration of the device itself and will not require external configuration, but this is environment specific.  A Network Admin and System Admin should be engaged for all activities dealing with network connections.

       

      Processes

       

      Each process in the diagram is either a .jar or .exe process.  This indicates if it is a JVM (Java Virtual Machine which is ran using the Java.exe process found in the Java Installation directory.) or if it is C based process.  Java processes rely on the Oracle Java Runtime to be install and operational.  C based processes require that certain .net libraries be installed and operational.  Please consult Oracle, Microsoft, and RedHat/IBM on any questions or concerns with these products, BMC does not support these products.

       

      These should also be kept up to date, just like the other products we've talked about, and some organizations might have policies where you have to monitor each of these processes to ensure their uptime or other key metrics.  Consult your company policies and experts for any tools you plan to use to monitor these processes.  (Like BMC TrueSight or Cisco Soalrwinds)

       

      Protocols

       

      The diagram above also details the different types of connections by coloring the connections to match the type of communication that is being done.  Remedy AR System Server uses RPC calls over TCP to communicate, but Tomcat uses HTTP(S) over TCP to communicate.  Each connection is unique, and would be secured in a different way.  At this point, it is just important to recognize the difference and understand which connections are used where.  We will discuss techniques to secure each type of connection in the next post!

       

      Action Items:

      1. Update the diagrams from 201 to include protocols, processes, and ports for each component you plan on utilizing in your system.
      2. Ensure that your charter includes roles and responsibilities for 3rd party products like Java, Tomcat, DB Server, and other components.
      3. Review and understand the products you're going to use.  This includes how to use Java Developer tools, SQL Developer tools, and other components that will be used for daily operations.
      4. Review and understand the upgrade paths and roadmaps for all the products you're going to use.  Apply this to your roles and responsibilities for operation and maintenance activities.
      5. Involve a System Admin, Network Admin, Security Operations, and a Database Administrator in your conversations!
      Share This:

      This is the second in a series of topics that will cover securing an entire Remedy ITSM system. This will be presented in several blog posts each meant to be read and reviewed in serial to each other.

       

      These sets of topics are meant to help guide a Remedy System Administrator into first understanding the potential threats to a properly configured Remedy ITSM environment. The goal is to introduce certain scenarios and facilitate conversation and policy creation, then eventually start you towards action items.  While this guide will attempt to cover many topics, your specific environment might have different business needs.  For these extra business needs, such as Government oversight or contractual requirements, you will need to consult in conjunction with a security specialist that both understands the Remedy environment as well the needs for your business.  While we will attempt to cover a large scope of items, including items outside of BMC’s control, it is inevitably up to the Remedy Administrator, System Administrator, Network Administrator, and the organization to understand the environment, plan the environment, and configure the environment for the implementation of a secure Remedy ITSM system.

       

      This will start with an overview of common industry standard security concepts.  Then we’ll review the questions that you should ask yourself and be able to answer to know what steps you need to take and to what degree do you need to implement these security requirements.  Also, we’ll review the specific Remedy products, and finally we’ll review the individual actions you can complete to improve the security of your environment.

       

      The next question most people would have at this point is “What can I do about it?”  And the answer is not as straight forward as we’d all like for it to be.  As we covered in post 101 - Concepts, the first thing to do is sit down and have a honest conversation with the data owners, product owners, stakeholders, and system administrators about what do you really feel is important in your system.  Once you've determined what you know and understand is important to your system, then you can use the list below to expand the conversation with your team to help determine what you need to protect and plan out your Remedy ITSM System. While this isn’t necessarily all the questions you need to ask, nor will this necessarily provide you answers or action items, it can be used to get a holistic view of the scope, scale, and design of your environment.

       

      1. What do we want to do with the Remedy AR System product?  (Help Desk, Asset Tracking, Service Request, etc.) - Once you determine what you’re going to use the system for, you can plan around it. Maybe firewall rules become more restrictive when you include contracts into it, or its much less restrictive when you just use it for Help Desk.
      2. What components of the ARS System do we want to utilize?  (Mid-Tier, RSSO, Smart Reporting, etc.) - No need to implement more features, or less features, than you plan on using for your company.  Typically, you want to utilize as few of the technologies as possible.  If everyone is going to use Smart IT, maybe you don’t make a Mid-Tier cluster.
      3. What is our current skill-set?  (Linux vs Windows, Oracle vs SQL, Tomcat vs IIS, etc.) - Use a technology that you, and your company, are comfortable with.  Don’t choose Oracle DB if you only have 1 person at your company that knows Oracle DB when there are 5 that know MS SQL.
      4. How big does the environment need to be?  (Check the BMC Recommend sizing) - Bigger means more complex.  And a system that is more complex than you’re able to handle can, and will, cause security vulnerabilities.  Don’t stand up 5 systems for just 50 end users.  A more complex system is much easier to “forget a patch” or to go unnoticed when there is a problem.
      5. How many different roles does my company have?  Which roles will be using Remedy?  What is the least amount of access they need to perform their duties? - This is one that might take some time.  Once you determine what you will use Remedy for, you’ll need to determine who will use Remedy.  If you’re implementing Service Desk only, then maybe you have a Remedy Admin, Remedy Developer, Service Desk Agent, Service Desk Manager, and Users roles. Where each one has different permission levels and license types.  This can impact cost of the system as well as peoples’ ability to view and change data in the system.  This might not always align with BMC’s default values and might include a many to many relationship between the two.
      6. All of the requirements of the system found in 101 - Concepts (contractual requirements, company policies, etc.) how do we implement those requirements?  - Turning functional requirements into technical achievements will be a critical part of the planning phase.  (ie, Functional requirement: secure communication between web browser and Mid-Tier.  Technical requirement: obtain and implement TLS 1.3 certificates for Tomcat process)
      7. What kind of security tools does my company have integrated into our platform?  How can we utilize these to keep our product secure? - Anti-malware software, firewalls (software-hardware), monitoring tools, etc.  Determine what tools you have at your disposal, determine what those tools can be used for, and how they can be configured to provide you the data/capabilities you need when you need them.

       

       

      Once you’ve answered these questions (and potentially other questions not mentioned here) you should be able to look at your environment and determine the general “layout” of the system as well as what things are important to you.  This can be done in tandem with the first post, although these are two entirely different sets of activities.  This will allow you to (in the upcoming sections) more quickly determine how you’d like to approach each one of these items.  After this process the Remedy Administrator should be able to answer questions about the system like “How much downtime is acceptable per month?”  “When will we have a open change window for maintenance activities?” and “What data in our system is consider Public, Private, and Personally Identifiable?”  These are known as functional requirements and these should be clearly stated in a charter of some kind, so the team always knows what they need to strive towards as well as sets expectations for any consumers of the application.  Without this, it is easy to lose sight of these requirements and can cause SLA failures, private information breaches, and other unnecessary risk.

       

      Action items:

      1. Create a charter that details the goals of your system.
      2. Start transitioning your functional requirements (from 101 - Concepts) into technical requirements.  (Do this in order of most important to least important)
      3. Create a diagram detailing the components of your system, including both present and future components.  This should include things like communication protocols, ports, servers, services/processes, and installed products.
      4. Assign the daily operations and maintenance actives for the system (from 101 - Concepts) to individuals or roles in the company.
      5. Involve a System Admin, Network Admin, Security Operations, and a Database Administrator in your conversations!
      Share This:

      This is the first in a series of topics that will cover securing an entire Remedy ITSM system. This will be presented in several blog posts each meant to be read and reviewed in serial to each other.

       

      These sets of topics are meant to help guide a Remedy System Administrator into first understanding the potential threats to a properly configured Remedy ITSM environment. The goal is to introduce certain scenarios and facilitate conversation and policy creation, then eventually start you towards action items.  While this guide will attempt to cover many topics, your specific environment might have different business needs.  For these extra business needs, such as Government oversight or contractual requirements, you will need to consult in conjunction with a security specialist that both understands the Remedy environment as well the needs for your business.  While we will attempt to cover a large scope of items, including items outside of BMC’s control, it is inevitably up to the Remedy Administrator, System Administrator, Network Administrator, and the organization to understand the environment, plan the environment, and configure the environment for the implementation of a secure Remedy ITSM system.

       

      This will start with an overview of common industry standard security concepts.  Then we’ll review the questions that you should ask yourself and be able to answer to know what steps you need to take and to what degree do you need to implement these security requirements.  Also, we’ll review the specific Remedy products, and finally we’ll review the individual actions you can complete to improve the security of your environment.

       

      The whole goal of IT Security is to find and mitigate vulnerabilities in a system.  This should be, at its core, the function of any IT Security policy that is implemented.

       

      First we need to understand "What is a vulnerability?" This is a way of gaining access of a system or data through a flaw.  The International Organization for Standards defines a vulnerability as “A weakness of asset, or group of assets, that can be exploited by one or more threats.”  Typically to a “lower level” of access than the person is assigned.  A way or method that can be used are called exploits.  There are several “levels” that exploits could occur at:

       

      1. Hardware – These are exploits that are executed at a HARDWARE level.  A popular example is the recent “Specter/Meltdown” bugs found in most modern CPU’s. (CVE-2017-5754) Work with your hardware vendor to help identify, understand, and remediate hardware level issues.
      2. Software – These are exploits that are executed at a SOFTWARE level.  A popular example is the “Heartbleed” vulnerability found with OpenSSL. (CVE-2014-0160) Work with the software vendor to help identify, understand, and remediate software level issues.
      3. Network – These are exploits that are executed at a NETWORK level.  The most common example of network exploits are open or improperly configured firewalls.  Work with the networking equipment vendor to help identify, understand, and remediate network level issues.
      4. Personal/Social – These are exploits that are executed by manipulating the people in the system.  An example is someone who doesn’t have access to a set of data, asking a friend to get the data for them.  A strong IT Security team should be able to help provide guidance, training, and policies to combat personal and social exploits.
      5. Physical – These are exploits that are executed by a physical means.  An example: someone who is freely able to walk into areas of a building where they don’t belong, which can expose them to information that they could use.  A strong IT Security team should be able to help provide guidance, training, and policies to combat physical exploits.
      6. Organizational/Policy – These are exploits that are usually performed by specific situations. An example is not having a continuity/contingency plan in place.  A strong IT Security team should be able to help provide guidance, training, and policies to combat organizational and policy exploits.

       

      After understanding the different ways can be used to exploit a system, next we need to understand what are common CAUSES of exploits?

       

      1. Complexity – Imagine deploying 100 web servers for an environment with only 50 people in it. This increases the complexity and increases risk un-necessarily.  What happens if you forget to patch one?  What happens if 1 gets compromised, but not all of the servers show symptoms?  The best solution to any "complexity" issues is to just have a plan for both present day and future implementations and actions for the system, and only focus on the systems needed to perform the task at hand.  Simplicity is key.
      2. Familiarity – Having familiarity with a system can be a good thing, but it can also provide an easy vector for an attacker due to the extensive knowledge and tools available to them.  The best way to combat this is to not divulge information about the system to anyone other than the people that need to know about the system in question.  Having a set list of duties the system will require, people that will fulfill them, and backup and contingency plans for labor are needed to help mitigate "familiarity" issues.  Implementing a system of least privilege would be a critical step to preventing familiarity exploits.
      3. Connectivity – This is caused due to physical connections being tampered with or excessive ports or protocols being opened or in use that aren’t necessary (among other potential issues).  To prevent connectivity exploits to a system, it is best to employ a network firewall and appropriately configure it to not allow traffic that hasn’t been predetermined to be safe.  (Whitelist/Blacklist)
      4. Management Flaws – These can also be called Policy Flaws.  This is where a scenario hadn’t been discussed or planned which allows (over time) exploits to become available.  The best way to prevent any management flaws is to have a policy in place for any scenarios and follow them.  Do you have an Access Policy, Upgrade Policy, Audit Policy, Monitoring Policies?  Then enforce these policies and (if needed) improve upon them.
      5. Design Flaws – Design flaws would include flaws to custom projects.  This could be using an old version of a tool or software, or not configuring a system appropriately.  Planning is the best way to overcome any design flaws.  You need to plan out the system based on the requirements needed, including system requirements, pre-requisites, implementation, maintenance, operational, and other benchmarks.
      6. Software Bugs – Software bugs are where there is a “hole” in the software itself where it might expose information to unwanted people.  The best thing to ensure software bugs are not a problem is to ensure you review and apply all updates to all software.  Also, reviewing your “Technology Roadmap” to ensure you’re in-line with the vendors support vision.  (Oracle Java LTS plan, RHEL LTS, etc.)  Understand the release cycle of your products and make plans and policies to ensure that you’re relativity up to date.
      7. Malware – This is software that is installed and works as designed, yet it (in some way) harms the system that it lives on and the intention (such as a process running on a system that is consuming system resources un-necessarily) Anti-Malware services will help with this.  Symantec, McAfee, and Kaspersky (and others) all of these popular vendors can help mitigate malware in a environment.
      8. Not learning from past mistakes! – You need to document, analyze, review, and plan for all previous mistakes.  Making a mistake is bad enough, don’t ignore it, learn from it.  This can also be expanded to learning from other’s mistakes, so you don’t make the same mistake!

       

      If you've made it this far, chances are you're at least interested in the topic at hand, or might even have a requirement to meet some of these goals in different ways.  But, eventually everyone is affected if there is a insecure system available.  It's not just high profile business like banks, credit card companies, or tech giants that are affected in big ways.  This can affect even smaller, seemingly less "critical" business like a simple grocery store chain or a movie studio:

       

      1. In November of 2014 a major Hollywood movie studio was compromised by all of their computer screens being hijacked.  Obviously causing a production loss for the company, but within days tons of their data ended up on file-sharing sites.  Including Social Security Numbers, passports, internal passwords, unpublished movie scripts, financial plans, and other information.  This was so devastating that the total cost cannot be calculated, and years of potential work was leaked to the competition.  This was a network exploit.  Some people consider this the single biggest data breach of all time due to the impact this had directly to their business.
      2. In 2013 a major US retailer had a breach of financial records that caused 100+ million of consumers credit cards to be exposed.  This cost the company over $300 MILLION of which their insurance only paid out $100 Million and it is becoming common practice to top out security related payouts to a maximum of $100 Million.  This was a physical/hardware exploit where a contractor was given unnecessary access to their credit card processing system.
      3. In 2013 a (formerly) major tech giant had a data breach that exposed 500 million users’ data. It came out later in the year, that the number was closer to 1 billion users.  Then in October 2017, the company dwindled so badly that they were sold and bought by another tech giant, who disclosed that the number was around 3 billion users.  This was a software exploit, and the original company’s brand name is virtually no longer recognizable, as they’ve been bought by another company for pennies on the dollar as compared to their earlier evaluation.

       

      None of the use cases above were compromised by a ITSM system flaw, but that doesn't mean there aren't some very critical data (maybe even more critical in some cases) floating around in ITSM systems out there, here are some examples of data that should be kept secure:

       

      1. What if someone was able to gain access to this weekends’ change calendar, and they knew about when you were going to reboot your firewalls due to a change?
      2. What if someone was able to gain access to your current list of assets, including the software installed as well as what version they’re running?  They could use this to cross reference back to known defects to infiltrate another potentially more critical system.
      3. What if someone was able to gain access to your people records and gain the location of each employee, including work at home addresses?  (That would include your address too!)
      4. What if someone was able to gain access to the list of contracts your company currently has, including their value, then sells the information to a competitor to help them win the next bid on a major contract?

       

      This list of questions is not meant to scare you into anything, but it is meant to help illustrate that a typical Remedy ITSM system does contain some sensitive information that needs to be protected in several different ways and it should not be taken lightly.

       

      So in summary, the first action item for any team setting up a secure Remedy ITSM System is going to be determining what is important to you.  If you have a list of requirements that have been provided to you from a regulatory body, then this step might be easy.  But if you’re being tasked with a open ended request to secure a ITSM environment, this might require quite a few days of thought.  You can not continue to any of the next steps unless you’ve done this step, by determining what is important to you, and to then determine where those vulnerabilities to that might be.  This is known as the identification phase.

       

      Action items:

      1. Make a list of all the daily operation and maintenance activities for the system.
      2. Create a list of items that your ITSM System is used for.
      3. Create a sub list for each of those items, that are sensitive data to your company.
      4. Review your companies security policies, ensure that you extract all requirements from your companies policies
      5. Review any regulations or contracts you might have that would govern your ITSM system, then extract those requirements as well.
      6. You might want to rank each of these action items from most important to least important.
      7. Involve a System Admin, Network Admin, Security Operations, and a Database Administrator in your conversations!

       

      Resources:

      https://en.wikipedia.org/wiki/Vulnerability_(computing)

      Publicly Available Standards

      CVE -CVE-2017-5754

      Heartbleed, The First Security Bug With A Cool Logo – TechCrunch

      https://www.tomsguide.com/us/pictures-story/872-worst-data-breaches.html#s2

      Share This:

      Hello everybody,

       

      I can't believe it has been almost 4 years since my last blog post. I haven't really had anything earth-shattering to share and/or some of the discoveries/tools I would have normally shared since my last post were lost in the shuffle of work and life.

       

      First a quick update... I have recently switch roles from customer to consultant. It was odd at first, not having a production system to take care of. I still leave my phone turned on by my bedside at night. Even though this is the first time in almost two decades I have not had official or implied oncall duties. The first week or two was also hard not having a Remedy system to touch and poke at. I now have a fresh install of ITSM 19.08 so I can once again reference an environment and get my Remedy fix.

       

      Traditionally I have worked for the same company years-on-end (13 year total at one company) and I am usually part of a small team (sometimes a team of one) that "owns" the Remedy system. I admit I have liked the stability, owning a system long-term, and the friendships built with teammates that go beyond work. However these situations also come with some limits. Maybe my choice to openly start engaging in this Community and the ARSlist before it, was driven by a desire to collaborate with people outside of my corporate silo?  (I am certainly not qualified to analyze that too deeply)  A mutual friend to many of us in this Community (his name rhymes with "KongBling", here is a picture of us together at WWRUG12) has had the opportunity to work as a member of larger teams and in a few more positions than I. Sometimes it seems like everybody in our little Remedy world has worked with this fellow at one point or another. I admit, I am a little jealous of that. I expect my new role will allow me to work with many more people than I have in the past and I am looking forward to this opportunity.

       

      Now for the goods... A peace offering for my lack of posting goodies to the Community the last few years... I was inspired recently by Adam Lawson sharing his PowerShell script How to Restart All of Remedy, With One Click!. I thought to myself "I have something similar but different that I could share with the Community": PowerShell Script - Service Restarter

       

      Both of our scripts do more or less the same thing but in different fashions. Adam's stops multiple Windows services on multiple hosts and them starts them back up while the script I created restarts multiple Windows services on multiple hosts in a rolling fashion. With a little modification his script could be used to easily bring down a whole Remedy environment and then start it back up again when needed (good for MS patching outage windows). My script is geared towards restarting services in a high availability scenario.

       

      I spent much of yesterday enhancing the script to accept host and service pairs as well as some code cleanup and commenting. Previously the script would only restart one service across multiple computers, for example AR Server or the Remedy Email Engine or Apache Tomcat. At my last organization I kept 3 different copies of the script, each configured to restart one of those services on the appropriate hosts. With yesterday's update the script can now handle restarting any combination of services and servers you configure. Of course I built this with Remedy in mind however you restart any Windows services you choose so feel free to share it with other system admins.

       

      Having options is a wonderful thing! Now you have scripts to stop then start all of your services at once or restart a bunch of services one at a time. I have never used PowerShell on Linux but I am hoping it would not take much to make Adam's or my script work on other OSes that can run PowerShell (a little reading made it look like Get-Service wasn't supported at least in earlier versions of PS Core for Linux).

       

      Relevant to:

      Remedy AR System

      Remedy ITSM

       

      Find Jason's other Tips and Tricks:

      Index of Jason's Quick Tip Blog Posts

      Share This:

      The Remedy Management Console can be used to manage server group operations. This console simplifies the server group management by providing a single location to perform many of the server group configuration related activities.

       

      The Remedy Management Console also known as the Server Group Administration Console was initially introduced in Remedy 9.1.04 (1802) as an unsupported version available from the BMC Communities. In Remedy 1805 release, a supported version was introduced.

      Key Benefits

      • Ease of monitoring operations and administration in a server group without having remote access to actual servers
      • Visibility into server queues for critical processes
      • One-click operation to start or stop server processes and enable or disable logs
      • Apply configuration settings consistently to all ARServers in a server group

       

      Introduction to different options on the Remedy Management Console

       

      Server Group Dashboard – 6 flashboards showing status on important ARServer activities for monitoring purposes. Shows number of records FTS indexing, CAI events, Application Pending, SYS:Action, Pending outgoing emails and Number of days for oldest email message. Please see Connect with Remedy webinar for more information on customization.

       

       

      Server group Configuration – Shows ARServer configurations same as Centralized Configuration form. Helps to manage all ARServer configurations of server group at one location. User can add, update configuration as it is allowed from Centralized Configuration form. You can use this interface to ensure that your settings are consistent across all the servers in the group. It can be also user to setup Global settings that can be set once and then adopted on all servers

       

       

      FTS Management – Allows you to view and update ARServer configurations for Full Text Index and Full text search. User can view and update FTS configuration for all ARServers in group. User can perform complete and form level re-indexing and monitor the status

       

       

       

      Manage Processes - User can manage different ARSystem processes without having remote access to actual host server. Form allows to perform stop, start, refresh, restart, add, update actions for specific process. This can be useful when it is required to make a change such as increasing Java Heap for plugin servers across the server group and you do not want to restart all the ARServers. A specific process can be modified and restarted without having whole ARServer restarted.

       

       

      Logs – Form helps to Activate and deactivate log files for each ARServer in group. Also allows to download log files from respective ARServer. Helps to capture longest running API and SQL statistics quickly. During troubleshooting logs can easily be enabled and disabled on multiple servers in the group.

       

       

       

      Manage User Licenses- View and manage server, application and bundled licenses in server group. This is the view same as License Review form except that it shows data from all the servers in the group at once. You can use this interface to see which users are consuming various licenses and allow to release active user sessions.

       

       

       

      Current Operation Owner – View ARServer name owning AR System operations in server group in real time. Useful while checking failover activity of specific operations in server group

       

       

      Ranking – Redirects to the AR System Server Group Operation Ranking form. User can manage operation ranking for each ARServer in group

       

      Failover – Redirects to the AR System Service Failover Ranking form. User can view service failover ranking for different operations. Service failover is used to manage Email Engine and Normalization operations

       

      Group – Redirects to the Group form. View, add, update all groups configured

       

      Users – Redirects to the User form. User can manage all Remedy users

       

      Configuration – Allow to customize environment such as customizing server group dashboard, setting configuration preferences, creating server name list, component list and settings list

       

       

      • Documentation

      Configuring and monitoring AR System server groups using the Remedy Management Console

       

      • Communities

      Remedy 9.1.04: Server Group Administration Console Overview Video

       

      • Webinar

      Connect with Remedy - Using the Server Group Dashboard Webinar Recorded Session - YouTube

      Share This:

      After you’ve enabled log forwarding, and you’ve identified your critical events needing to be captured, the next phase is to use the Splunk application to extract usable fields so you can sort, report, search on.  This is a quick summary on how to extract some basic fields from the armonitor.log (which is used to monitor the Remedy server processes, including AR System Server). This assumes that you’ve set up Forwarding and Collecting Data from the previous article.

       

      Configuring Source Types

       

      1. Log into the Splunk Enterprise Administration Console


      2. Click “Settings -> Source Type” to open the “Source Type” console.


      3. Click on the Source type that was created with the Add Data Wizard.  In this case it was “AR Monitor Log”.


      4. In the Edit Source Type console, you’ll want to configure a few items, which include event breaks, gathering time stamps, and the categories.  Select the Category as needed and if you need any Indexed Extractions.


      5. Event Breaks are the first option that will need to be selected.  You can choose either Every Line or Regex, based on your requirements.


      6. The next thing to configure is the Timestamps.  It is recommended that you set the timestamp configuration to “Advanced” and configure it as below.

      Timezone

      Set to AR System Server Time Zone

      Timestamp Format

      %a %b %d %Y %H:%M:%S.%Q

      Timestamp Prefix

      /.

      Lookahead

      180

      7. Once that has been configured, you can click “Save”.  This has Configured the source type so it will start to parse the logs correctly.

       

       

      Configuring Field Extractions

       

      1. To configure field extractions click “Settings -> Fields” to open the “Fields” console.


      2. Then click “Field extractions” to view and edit field extractions.


      3. Click “New Field Extraction” in the top right-hand corner to open up the New Field Extraction wizard.


      4. You can now configure the extraction how you’d like, for this example we’re going to create the extraction in the “Search” application, give it a name of “AR Monitor Extraction” and apply it to the REM 1908 Source.  This would be considered a “inline” extraction.  Then you’ll need to put in your expression to extract the fields. See below for a example expression to extract AR Monitor logs.

      <(?P<Type>\w+)> <TNAME: (?P<Thread>[^>]*)\s*> <(?P<LogLevel>\w+)\s*> <(?P<Class>[^>]*)> <\s*(?P<ClassLine>[^>]*)> .................................... (?P<Details>.*)

      5. Once you’ve configured this, Save the new extraction.  You can now modify the permissions as needed, contact your Splunk Admin if you have questions about permission.  Once you’ve done this, you should be able to open up a search and start reviewing the index created earlier to view the data needed.

       

       

      Once you’ve been able to perform these actions, you can start creating Dashboards, Alerts, and other Splunk activities.  If you need help with this, then please contact your Splunk Admin for more information.  If for any reason the Splunk Server is not working, please contact Splunk as this is their application.  BMC will not provide support for this product and this documentation is provided AS IS, as a courtesy.

      Adam Lawson

      Log Forwarding With Splunk

      Posted by Adam Lawson Employee Jan 10, 2020
      Share This:

      Splunk is a Enterprise Application that will collect data from different sources, and aggregate them under one console allowing for a more complete knowledge when troubleshooting or analyzing environments for problems.  Splunk has the ability to ingest several log types, including Windows Server Event logs, Linux System logs, and application logs.  BMC Remedy products have their own logs that are logged to files found on the application server independent of the system logs itself. Because of this, there are additional steps that might be needed as compared to other systems to allow Splunk to consume BMC Remedy logs.  Splunk provides all of the tools needed to do this, and BMC does not provide support to execute any of these actions.

      The following steps are some high-level steps that you can perform to collect logs from your BMC Remedy servers and push them into Splunk.  If there are any questions about these steps first please view Splunk’s Universal Forwarder and Splunk Enterprise documentation (link below), or contact Splunk for more information.

      https://docs.splunk.com/Documentation/Forwarder/latest/Forwarder/HowtoforwarddatatoSplunkEnterprise

      Configure Splunk to Receive Logs (Enable a receiver)

       

      1. Log into the Splunk Enterprise Administration Console

      2. Click on “Settings -> Forwarding and Receiving”

      3. Click “Add New” Under the Receive Data section.

      4. Provide a port to receive the logs with, click “Save”

      5. It will then bring up the Receive summary page.  You can add a new receiver here and review others.  (In a typical Production Environment, you’ll have several Receive ports configured.  Consult your Splunk Administrator if you have questions about which one to use)

       

      Splunk Enterprise is now ready to receive data from a new Splunk Forwarder.  If for any reason the Splunk Server is not working, please contact Splunk as this is their application.  BMC will not provide support for this product and this documentation is provided AS IS, as a courtesy.

       

       

      Install Splunk Forwarder On Same Server As Remedy AR System Server

       

      1. Start Splunk Forwarder Installer.  Read and Accept the License Agreement from Splunk.  Choose if you’re using Splunk On-Premise or Splunk Cloud and click “Customize Options”

      2. Choose install location and click “Next”.

      3. Configure the SSL Certificates based on your organizations configuration, click “Next” to continue. If you have questions, ask your Splunk Admin.

      4. Select the type of account to use by the Splunk Forwarder application.

      5. Choose what data you want to be sent to the Splunk Server by the Forwarder.  For the purposes of Remedy AR System, you’ll need to select the DIRECTORY of your “DB” folder. After this is configured, click “Next”.

      6. Configure the Splunk Administrator account, click “Next”.

      7. Configured your Deployment Server.  This should be the Splunk Server’s configuration process.  If you don’t know this information, contact your Splunk Administrator. Click “Next” to continue.

      8. Configure your Splunk Forwarder to send the data to the Receiver we created before.  Click “Next” to continue.

      9. Click “Install” to proceed with the installation of the Splunk Forwarder.

      10. Once the installation is complete, the forwarder is configured and running as needed for a Remedy AR System Server instance.  Click “Finish” to end the installation.

       

      If for any reason the forwarder is not working, please contact Splunk as this is their utility. BMC will not provide support for this product and this documentation is provided AS IS, as a courtesy.

      Consume Remedy Logs in Splunk

       

      1. Log into the Splunk Enterprise Administration Console


      2. Click “Settings -> Add Data” to open the “Add Data” wizard.


      3. Choose the method to gather logs from.  To collect Logs from Remedy AR System Server, use a Splunk Forwarder by clicking “Forward” from the “Or get data in with the following methods” section.


      4. On the “Select Forwarders” section, choose the forwarder that you want to collect, and give it a New Server Class Name.  In this example, we’re going to Select the server “clm-aus-tt8f8w” and then give the new name of “REM1908” as this is a Remedy 19.08 System.  Click “Next” to move to the next section.  You can obviously choose the Server Class name as whatever you want it to be, just make sure it is relevant to your scenario and that you can easily remember and use it.


      5. Choose the source as “Files & Directories”, and choose the file you want to consume. In this example, we’re going to consume the armonitor.log file.  You can choose to setup a whitelist or blacklist if desired.  Click “Next” to continue to the next section.


      6. Choose the Source Type as a “New” option and then fill out the source information as needed, if you have questions about what to select contact your Splunk Admin. Then choose a Index for use with this application.  If you need to create one, contact your Splunk Administrator.  Click “Review” to continue to the next section.


      7. Once you review and confirm the settings, click “Submit” to finish the log collection.


      8. Once you’ve clicked “Submit” it will start indexing the logs.  This might take some time the first time you do it. Additionally, you can start to groom the data to make better use of it.

      This includes Extracting Fields, adding additional data, and building Dashboards.  You can do all of these from the final page.

      Splunk Enterprise is now receiving data from the Splunk Forwarder installed on the Remedy Server. If for any reason the Splunk Server is not working, please contact Splunk as this is their application.  BMC will not provide support for this product and this documentation is provided AS IS, as a courtesy.

      Filter Blog

      By date:
      By tag: