Remedy 9 is a new release from BMC for enterprise service management platform: The fresh user interface is built on a mobile first design with focus on improving productivity. This new release is supported by a strong platform built on Java open source technology stack.
In addition to features such as support for mobile, intuitive self-service applications, adaptive reporting, and enhanced productivity with formless data entry, Remedy 9 also has better performance, vertical and horizontal scalability, and high availability.
BMC Remedy applications focus on more than just the response time or the rate of operations, they provide an intuitive, people centric user experience with optimized navigation for higher productivity, a flexible architecture, and an easy centralized configuration to support complex deployments and enterprise workloads.
To set a new benchmark demonstrating the unparalleled performance and scale of Remedy 9, the BMC Performance Team collaborated with Oracle labs to set up an enterprise-level deployment using Oracle Solaris and SPARC servers while using Micro Focus’s SilkPerformer to simulate the complex workload representing real world usage.
We want to share in this blog the results of a 10,000 concurrent users’ workload on BMC Remedy ITSM Suite 9.0 SP1 using Oracle 12c database with large data volume, running on Oracle SPARC T5 servers with Oracle Solaris 11.
For accuracy, we modeled the workload after a typical “a day in life” scenario at an enterprise-level deployment consisting of:
- 10K concurrent users executing ITSM/SRM and Smart Reporting use cases using standardized workload.
- Atrium batch jobs, Normalization Engine (NE) and Reconciliation Engine (RE) running in continuous mode processing 100,000 configuration items (CI).
A natural consequence of this joint effort is the establishing of the benchmark metrics for customer reference in their enterprise deployments.
This blog summarizes the result of the 10K benchmark. A whitepaper will be published with complete workload details (how many user types, the use case execution rate, the number of entries created in the DB, data volume for the workload, etc.), the architecture, the hardware, configurations, performance tunings, and optimizations.
Once the whitepaper is completed, we will update this blog with the link to the whitepaper.
Update: Here's the link to complete benchmark report:
For a benchmark of this magnitude, we collaborated with the Oracle Lab team for the hardware, configurations, and deployment. We also needed Micro Focus’s support for the load tools and the 10K virtual users licensing.
The sub-sections below describe the BMC product combination as tested, the Oracle hardware used, and the load testing tool and paradigm.
The ITSM Stack
We benchmarked the ITSM Product Suite V. 9.0SP1. The benchmark consisted of the following product combination: BMC Remedy ITSM Suite, BMC Service Request Management, BMC Knowledge Management, BMC Atrium Configuration Management Database (BMC Atrium CMDB), and BMC Smart Reporting.
Oracle Lab Hardware
For this benchmark, the complete server-side environment was deployed on VM’s hosted on a single SPARC T5-8 (1024 GB RAM; 8x 3.6 GHz SPARC T5) with the Oracle 12C database deployed on a single SPARC T5-2 (512 GB RAM; 2x 3.6 GHz SPARC T5). The BMC AR Platform was deployed as: 5 Mid-Tier VM’s, 1 Smart Reporting VM, 1 AR Integration Server VM, and 5 AR Server VM’s (all 6 AR servers configured as an AR Server Group).
As part of the environment preparation, we executed the workload with HTTP and with HTTPS and compared the results. The SSL layer was handled by the Apache Tomcat 7.0 deployed on JDK1.8.0_45. The average use case response times between HTTP and HTTPS were comparable, i.e., only a negligible increase in use case times that could have been due to the margin of network variation or load execution.
Additionally, the average CPU usage on the VM's hosting the Mid-Tiers were up only slightly, just over 1.5% for the duration of the workload execution. This is because the SPARC processors have native hardware encryption and this hardware encryption layer is accessible to Tomcat via the Oracle JVM implementation.
Lastly, note that for the 10K users workload, we used less than half of the SPARC T5-8’s resources.
BMC had worked with Micro Focus to develop the AR Web plugin module for Silk Performer to make use case scripting and variable parameterizing trivial (the Mid-Tier uses a special URL protocol for variable serialization which makes use case scripting difficult without this Silk Performer plugin).
In our 10K benchmark, we used the Micro Focus Silk Performer as the load driver to simulate the concurrent online users. Each user, after login, was issued a set of unique cookies by the server. Silk Performer uses these distinct sets of cookies to simulate the online users. The server cannot distinguish the simulated 10K concurrent users from real users.
To achieve the performance level as captured by the result below, we made many optimizations. We provide a hi-level summary here:
- With large data volume, the key to getting optimal performance is understanding the database behavior. For this benchmark, we fine-tuned the Oracle 12C database and also created several additional indices.
- Remedy 9 is 100% Java, understanding the behavior of the JVM, especially garbage collection and heap usage, was another key to getting optimal performance. We used the low-pause GC for the service Mid-Tiers and the G1GC for the reporting server as well as adjusted the heap ratios.
We have included all our findings and configurations in the full whitepaper.
The 10K Benchmark Result Summary
The workload consisted of 10,000 online users executing the ITSM/SRM use cases while a batch process creating 10,000 new configuration items (CIs) and relationships, and updating 90,000 existing CIs with normalization and reconciliation running in continuous mode.
Running normalization and reconciliation processes in continuous mode provided near real-time reconciliation of CIs. In this mode, the reconciliation and normalization engines run continuously to normalize and reconcile CIs in small batches based on time interval or record count configuration settings.
CIs and relationships are reconciled into the BMC.Asset dataset, which has approximately 5 million CIs and relationships of prepopulated data for this workload. The source dataset, BMC.Oracle, also has prepopulated data of approximately 5 million CIs and relationships.
Use Case Metrics
The following chart shows the Micro Focus Silk Performer response times:
The following table summarizes the entries created during the 10K workload for a duration of 1 hour:
Number of entries
New Incidents Created by support
Existing Incidents Resolved by support
New Service requests Created by end users
Service Requests updated with Activity Notes
Incidents by web service
Incidents work log modified by inbound emails
Changes modified to Closure
Work orders updated
Total emails messages Inbound & Outbound
CMDB - CIs & relationships created and merged onto BMC.ASSET
CMDB - CIs updated and merged onto BMC.ASSET data set
Smart Reporting - Accesses the Incident Crosstab report, and saves it
Smart Reporting - Accesses Incident Management Dashboards with various drilldown
Smart Reporting - Custom Report Creation with 10 various Fields
- The Remedy System is a configurable platform. It is imperative to configure and tune the platform depending on your workload as well as deployment infrastructure. With a properly configured environment, the ITSM Product Suite V. 9.0SP1 can scale for large, enterprise-level deployments.
- The ITSM Product Suite V. 9.0SP1 scales vertically and horizontally.
- The performance in the form of use case response times are comparable and in most cases, better than previous versions.
- The Smart Report is heavily dependent on the database performance and can benefit from database optimizations.
- At 10K concurrent users workload, we used less than half of the resources of the SPARC T5-8, so a single T5-8 can scale to potentially a larger users workload (we did not test this due to limited Silk Performer virtual user licenses).
- On the SPARC T5 chipset, the switch from HTTP to HTTPS with the SSL layer handled by Tomcat, introduced only a negligible variation in use case times (attributable to network margin of error) and less than 1.5% average increase in Mid-Tier CPU for the workload run duration.