- by Joe Goldberg, Lead Technical Marketing Consultant, BMC Software Inc.
As we move into the second layer of the automation stack we encounter the convergence of business and IT processing that is one of the drivers behind the “Workload” terminology that has been adopted for what previously was job scheduling. The workload we are managing is expanding beyond traditional application boundaries and is interacting with transactional and real-time IT environments.
Service Level Management for batch business workload can be a topic of a separate post but at a minimum, there must be a way to define and express service levels or deadlines, a business oriented name for the service and actions to be taken if an SLA event is detected. Once defined, the workload automation solution must be able to proactively monitor progress towards SLA fulfillment. Once a potential breach is detected it is immensely helpful to have visualization and simulation facilities that help you to identify the source of the potential delay and to take corrective action.
Batch workload does not operate in a vacuum. The batch services we have been discussing are usually components of business services. Let’s consider a financial services organization. Interactive banking services such as deposits and funds transfers are usually finalized in batch. This is why a transaction made “after close of business”, for example, does not appear in your account until the following day. If financial account reconciliation or inter-bank transfers fail to process, web-based users may not be allowed to perform transactions because their account balances may not be correct. In order to visualize these relationships to put Batch Services into proper context, IT Service Modeling and Impact Analysis must incorporate batch SLAs. This visualization is enabled via the CMDB. The CIs and their relationships form the Service Model.
Let’s skip ahead to the topic we really want to discuss today; Dynamic Workload Management.
The dynamic aspects of today’s workload relate to workload itself as well as the environment in which it operates. As the importance and relevance of workload automation continues to permeate the organization, non-traditional users such as financial analysts, business unit managers, and application owners are finding they must apply workload automation in order to do their jobs. These users frequently submit analytics and other ad-hoc jobs. This translates into a larger, more volatile workload than has existed before.
From the IT perspective, virtualization and cloud technologies have made infrastructure less rigid than in the past. Organizations are seeking cost optimization benefits by configuring application resources for average usage with the intent of scaling up or down as required, based on the demand.
These two factors, random workload and on-demand infrastructure, combine to create the need for dynamic workload management.
Dynamic Workload Management introduces several capabilities to help organizations deal with unpredictable workload while maximizing the benefits of elastic infrastructure. The first is a policy-based approach to managing incoming work. Instead of designing workload to operate at specific times, workload policies categorize workload and define rules that are applied whenever those certain categories of work arrive. For example, when analytics jobs are submitted by the Finance department between 9 to 5 (high resource utilization periods), only a small number of those jobs can run concurrently. However, if it is financial quarter end, Finance analytics get close to top priority and we run these jobs as quickly as possible with much higher limits on concurrency. A policy-based approach lets us pre-define how we want to handle the workload mix and then allows the technology to apply these rules automatically.
Similarly, we can define how resources are made available to different categories of workload. It has been a good practice for quite some time to avoid using explicit hostnames when defining where workload runs. This mechanism is called node grouping. A logical name is assigned to a collection of server resources and workload is bound to that logical node group. Dynamic Workload Management extends this concept inseveral ways.
First, the node group on which workload will execute is now determined by policies. Using the Finance example from above, we can send work to a small node group when we are constraining the workload but to a larger and more powerful collection of servers during financial quarter end.
Second, the actual resources that make up a node group can vary dynamically. We can add either physical or virtual servers, including cloud-provisioned servers. We can also re-purpose physical servers. Using the classic example of a web or database server that supports a real-time application nine to five (if anyone still has such applications), we can define participation rules that add our database server to the workload environment during its idle time so that we can make productive use of what otherwise would be a wasted resource. Agentless Scheduling and dynamic nodegroups make this process significantly easier.
Third, the jobs that are submitted must become independent of specific hosts. Traditionally, jobs are tightly bound to a small group of machines. If we then wish to re-direct such jobs elsewhere, it becomes a complex, manual effort. To gain the flexibility required, the workload automation solution must directly manage the control language (scripts, JCL, etc.). This capability is called embedded scripts and allows jobs to be sent to any available machine, including a dynamically provisioned one that did not exist until moments ago, without concern about availability of scripts on that target host.
In conclusion, Dynamic Workload Management can be described as the combination of Workload Virtualization which allows us to manage our batch workload in a way that makes it independent of the IT infrastructure in which it must run and an elastic infrastructure that is itself also independent of physical hardware constraints.
The postings in this blog are my own and do not necessarily reflect the opinion or position of BMC Software Inc.