This is the third in a series of topics that will cover securing an entire Remedy ITSM system. This will be presented in several blog posts each meant to be read and reviewed in serial to each other.
These sets of topics are meant to help guide a Remedy System Administrator into first understanding the potential threats to a properly configured Remedy ITSM environment. The goal is to introduce certain scenarios and facilitate conversation and policy creation, then eventually start you towards action items. While this guide will attempt to cover many topics, your specific environment might have different business needs. For these extra business needs, such as Government oversight or contractual requirements, you will need to consult in conjunction with a security specialist that both understands the Remedy environment as well the needs for your business. While we will attempt to cover a large scope of items, including items outside of BMC’s control, it is inevitably up to the Remedy Administrator, System Administrator, Network Administrator, and the organization to understand the environment, plan the environment, and configure the environment for the implementation of a secure Remedy ITSM system.
This will start with an overview of common industry standard security concepts. Then we’ll review the questions that you should ask yourself and be able to answer to know what steps you need to take and to what degree do you need to implement these security requirements. Also, we’ll review the specific Remedy products, and finally we’ll review the individual actions you can complete to improve the security of your environment.
Up until now we've been reviewing concepts and planning out our approach to securing our environment. All of the steps up we've discussed up to this point, could really be applied to virtually any system you'd implement in any company, but this outline is going to get more into the specific for the Remedy ITSM system itself, and how it might be different from other systems. After reviewing this, you might find yourself going back and changing your original requirements. This is not only a good thing, it is expected. Your system requirements should be fluid and ever changing as your needs and understanding of the product changes.
Now that we have a clear understanding of what our system will look like, what purpose it serves, and what kind of requirements we have, we’ll need to now understand the products that we’re getting ready to use, their dependencies, and the roles and responsibilities needed to properly administer the system for day to day operations and long term success. The first thing that will need to be accounted for are updates on the Remedy products, this should have been one of the first things you came up with in 101 - Concepts and 201 - Planning during the assigning Operation and Maintenance actives.
There are 4 types of "updates" the Remedy product will have:
- Release - These are MAJOR version changes of the Remedy products. Remedy 9 is the latest major release.
- Version - These are MINOR releases of the Remedy products. The latest version at the time of writing this is 20.02. This is read in a “YearYear.MonthMonth” cadence to help you signify the version number to the actual release date to better understand and track upgrade opportunities. (Version 20.02 is still in the Release 9 family.)
- Patch – Patches are released on a regular cadence for each version. These will include fixes both functional and security as well as some new features. It is always recommended for you to be on the latest patch level. The latest patch was 19.02 Patch 1 (19.08 and 20.02 has no patches as of the time of the writing), it can also be known as “19.05” as it was released in May, but it is officially referred to as 19.02 Patch 1. At this time, 19.08 and 20.02 does not have any patches planned for the future, all fixes will be addressed in hotfixes.
- Hotfix – Hotfixes are released on a as needed basis for each version/patch. These will include fixes both functional and security and will not add new features. It is always recommended for you to be on the latest hotfix level. New hotfixes (typically) will only be released for the most recent patch of any currently supported version. These are updated regularly and should be checked often.
But this bring us to how to check for the newest versions, patches, and hotfixes? This is different depending on which type of update you're looking for.
Releases and Version
Check the BMC Doc’s page for Remedy Platform to see if there is a new version listed. If found, there will be instructions for “upgrading” found in the Doc’s page for that version.
Check the BMC Doc’s page for the Remedy Platform Version that you currently have installed under “Release Notes and Notices”. This will include any notices for this version (both security and functional) as well as any notices for patches. The patch will include instructions for applying the patch. https://docs.bmc.com/docs/display/ars1902/Release+notes+and+notices
Hotfixes can be found for all AR System products at the link below. They will come with their own set of instructions to follow. ftp://ftp.bmc.com/pub/ARRecommendedFixes/
This is a general outline, and is mostly true for all Remedy products including, but not limited to, AR System Server, Atrium CMDB, Atrium Integrator, ITSM Suite, Service Request Management, and Service Level Management. Each product will have their own documentation page, downloads, and procedure for apply the versions, patches, and hotfixes.
These can all be checked for manually. Additionally, these can always be “subscribed” to by clicking the link below:
By doing so, you will be alerted by email any time there is a product update which includes major security vulnerabilities, patches, and new versions.
It is important to always be on the latest version of a software that you can possibly support, as this will often be the only way to mitigate any vulnerabilities that are found. Not to include other functional and performance improvements that we're able to make along the way.
You can learn more about Remedy Version strings at the following Communities post which is updated regularly as new version and patches come out. This document will not be updated and is not the place to discuss specific issues or questions with the version strings, just to educate you on the granularity so you can better understand how to maintain your system.
There are other dependencies that you might have in the Remedy ITSM system that will also require you to do regular maintenance task. Here is a list of a few common dependencies and a few key highlights that have been gathered for you. You should ALWAYS consult the vendor of each of these products with any questions in regards to licensing, usage, installing, patches, and other operations and maintenance questions you might have. BMC can not direct you on how to manage these components for your environment, the information below is provided as is and contains no implied warranty or declaration of support.
- Overview - Java SE | Oracle Technology Network | Oracle
- Licensing - The current version of Java - Java SE 11 is available from Oracle under an open source license at JDK Builds from Oracle . Java SE 8 remains free of charge for general purpose desktop and server use and is available under the Oracle Binary Code License (BCL) at Java SE - Downloads | Oracle Technology Network | Oracle . https://bmcsites.force.com/casemgmt/sc_KnowledgeArticle?sfdcid=kA01O0000015FLhSAM
- Oracle Roadmap - Oracle Java SE Support Roadmap Java has recently changed their schedule for releases as well as supportability. It is VERY important that you understand this so you can ensure that you’re not vulnerable for certain attacks. Oracle has a open source option, LTS options, non-LTS options, feature based releases (GA) and security based releases (BPR). You should contact Oracle to better understand all of these factors in your decision for which release, version, and your upgrade/installation path and plan.
- Remedy must have a defined Java Instance. This requires the java path to be absolute. Best practice is to put Java in a “common” or “generic” folder such as D:/Oracle/Java/JRE or etc/Oracle/Java. This allows you to manage the path more easily, as the Oracle Installer naturally puts Java in a specific path that matches its version number. This change to the Java installation makes the upgrade process much less painful, allowing you to more quickly perform regular updates.
- Where to go - Java SE - Downloads | Oracle Technology Network | Oracle
- Overview - Apache Tomcat® - Welcome!
- Tomcat is a open source Java Servlet. It is developed and available on the Apache License version 2. Licenses
- You can learn more about the technology and roadmaps here: Apache Tomcat® - Which Version Do I Want?
- The Remedy Mid-Tier comes with a installation of Tomcat for your use in a development or non-production system. BMC does not recommend using the bundled Tomcat installation in a production or secure environment. You should install Tomcat separately to the Mid-Tier component and should be installed before you do the BMC Mid-Tier, Smart IT, Digital Workplace, or Remedy Single Sign-On products. There is a documentation portal for the version you plan on using on the Apache Tomcat homepage: Apache Tomcat® - Welcome!
- Just like the Java installation, installing Tomcat to a “common” or “generic” folder location will allow for easier upgrades of the Tomcat application for more quickly and regular updates.
This should always be handled by a SQL Administrator. Oracle DB and Microsoft SQL Server are their own intimate parts and are constantly changing. Please consult the BMC documentation to determine product compatibility and chose your version based on this information. Contact Microsoft and Oracle for more information on these products and how to support them. BMC Product Availability and Compatibility
This should always be handled by a System Administrator. Red Hat/IBM Linux and Microsoft Windows Server are their own intimate parts and are constantly changing. Please consult the BMC documentation to determine product compatibility and chose your version based on this information. Contact Microsoft and IBM/Red Hat for more information on these products and how to support them. BMC Product Availability and Compatibility
Keeping all of the software components above up to date is the primary way to fight against software defects that result in vulnerabilities. As time goes on, vulnerabilities are discovered and corrected. This happens at the hardware, system, and software levels. Having a plan in place to ensure you're running the latest software version possible should be a high priority.
Understanding Remedy ITSM
The next item that we need to understand is the Remedy AR System Architecture itself. This can be very overwhelming at first, but is something that you need to break down one item at a time. The below diagram will outline a network diagram of a single server ITSM system and all the processes that it would have, your environment WILL vary. Each component is detailed as a separate "server" in the diagram, but one single hardware server can host many of these processes (like in a development environment) or they can be spaced out infinity (like in a high availability production environment).
Note: The above diagram will be replaced with a interactive version in the future. For now, it might be best to download this image locally, and view in a application that will allow you to zoom in and out, as there are several components that have text next to them that will be helpful.
Remedy ITSM Connections
Here are some key takeaways from the diagram above:
- Everything starts with AR System Server. This should be the first component installed, upgraded, patched, tested, and monitored in any Remedy ITSM environment. All connections either start with AR System Server or start with the goal of eventually reaching AR System Server and is the center of a ITSM system.
- All the plugins start up with Remedy AR System Monitor, which is configurable by the armonitor.cfg file. In a typical environment, only Email and Flashboard plugins are started separate of AR System Server (Windows).
- Each server has its own configuration and logs. While they also share some configurations. This is defined on a per server basis and is not a common theme that you can always rely on. The diagram above includes the name of the process that is running (what you can see in task manager) along with any relevant configuration files and the default port setting or the setting itself you can change to manage the port numbers. All settings should be managed using the Centralized Configuration form if possible. Centralized configuration - Documentation for Remedy Action Request System 20.02 - BMC Documentation
- The communication between each plugin server and AR Server is critical. And this communication must work for the whole system to work. Firewall rules may need to be implemented to allow this communication. Even if you don’t use some of these components, like FTS, this communication should still be available to the server it can be used by other components. All of these ports are configurable.
- Client components (like Mid-Tier, Developer tools, and more) are all started up separately of AR System Server and will tend to error out UNLESS AR System Server is up and running and is available for communication. This is why it is important to ensure AR System Server is set up and working first. These are also configurable ports. But, it must match the settings configured for AR System Server.
- This diagram is to be used as a outline for you to build your own architecture. It does not consider things like a load balancer, custom plugins, email servers, and other items. This diagram was built with the intention to mimic a single server environment for a full ITSM System, so your environment will more than likely be even more complex and might have many other requirements.
External connections often go out onto the network and might cross several other systems before they end at their destination. This includes LDAP servers used for authentication, third party web services, and even database integrations like Atrium Integrator connections. But this also will include connections like Mid-Tier or Smart IT to AR System Server when they’re on different systems (recommended configuration for a production system). These will often require not just local configuration, but also configuration of additional network devices like routers and firewall appliances. A Network Admin and System Admin should be engaged for all activities dealing with network connections.
Internal connections, almost always, stay locally and “loopback” to the same server. Usually, these will include things like the plugin servers. Almost always these connections only require a local configuration of the device itself and will not require external configuration, but this is environment specific. A Network Admin and System Admin should be engaged for all activities dealing with network connections.
Each process in the diagram is either a .jar or .exe process. This indicates if it is a JVM (Java Virtual Machine which is ran using the Java.exe process found in the Java Installation directory.) or if it is C based process. Java processes rely on the Oracle Java Runtime to be install and operational. C based processes require that certain .net libraries be installed and operational. Please consult Oracle, Microsoft, and RedHat/IBM on any questions or concerns with these products, BMC does not support these products.
These should also be kept up to date, just like the other products we've talked about, and some organizations might have policies where you have to monitor each of these processes to ensure their uptime or other key metrics. Consult your company policies and experts for any tools you plan to use to monitor these processes. (Like BMC TrueSight or Cisco Soalrwinds)
The diagram above also details the different types of connections by coloring the connections to match the type of communication that is being done. Remedy AR System Server uses RPC calls over TCP to communicate, but Tomcat uses HTTP(S) over TCP to communicate. Each connection is unique, and would be secured in a different way. At this point, it is just important to recognize the difference and understand which connections are used where. We will discuss techniques to secure each type of connection in the next post!
- Update the diagrams from 201 to include protocols, processes, and ports for each component you plan on utilizing in your system.
- Ensure that your charter includes roles and responsibilities for 3rd party products like Java, Tomcat, DB Server, and other components.
- Review and understand the products you're going to use. This includes how to use Java Developer tools, SQL Developer tools, and other components that will be used for daily operations.
- Review and understand the upgrade paths and roadmaps for all the products you're going to use. Apply this to your roles and responsibilities for operation and maintenance activities.
- Involve a System Admin, Network Admin, Security Operations, and a Database Administrator in your conversations!