It’s been a few weeks now, but the Heartbleed OpenSSL bug has not gone away. The bug itself is not that hard to fix. Patches for affected operating systems and libraries were available on the same day as the announcement.
However, recent survey data from Netcraft (as reported by re/code) indicate that the fires are still smouldering, with only 14% of sites having completed all the required steps to fix the Heartbleed bug.
Traditionally, IT professionals divide their work into “keeping the lights on” and “firefighting”. Fixing the Heartbleed bug definitely fell under the second category. Fighting fires means lots of people running around, screaming, confusion, and elevated stress levels and mess all around. Because of the confusion, it's not the most efficient way to get things done, but most of the time it works - eventually.
But what happens once the fires are out and everyone goes back to concentrating on keeping the lights on?
Six Steps to Fix Any Configuration Problem - Not Just Heartbleed
What is a good process to put into place to make sure that problems like Heartbleed don’t crop up all over again once the IT team have taken their eye off the immediate issue?
- Schedule regular discovery across the entire IT estate
- Deploy management agents to any previously unknown servers
- Execute audit of all discovered systems to identify security and compliance violations
- Review dependencies and impact of remediation, set exceptions as necessary
- Approve and schedule remediation
- Update documentation trail automatically
Automated discovery is a must. The IT department can no longer rely on being the sole source of IT resources. It follows that their list of provisioned systems is no longer an exhaustive list of all systems, and therefore a system is needed to discover systems provisioned through parallel, “shadow IT” mechanisms.
This means that the old security paradigm that divides the world into “inside the firewall” and “outside the firewall”, with maybe a DMZ in between, is no longer valid. Systems way off in the public cloud may still be logically “inside” the network. This means that compliance and security audits can no longer focus on a short list of “core” systems. Any and all systems are potentially “core” in terms of vulnerability and dependency, and need to be audited accordingly.
Because more and more systems are being provisioned without input from the IT department, IT can no longer rely on controlling the templates used for provisioning or the definitive media library used to deploy applications. The assumption has to be that there will be systems provisioned with obsolete or otherwise vulnerable configurations, and will have to be updated in the field.
The democratization of IT means that the owners of those systems may well not have the skills to harden them on their own, so IT will need a mechanism to distribute, validate and enforce secure configurations. Because of the rate of change of virtualized environments, this process has to be automated to keep up with the current state of IT.
Finally, once the fires have been put out and everyone has moved on to the next issue, people will immediately start forgetting the details of Heartbleed and all of the things they had to do to fix it. An audit trail is needed to keep that historical memory, but nobody has time to create one of those by hand while also fighting fires. The documentation needs to be created automatically for every action.
None of these requirements – automated discovery, and automated configuration management – are new. Vulnerabilities like Heartbleed simply serve to underline the urgency of fulfilling them, to avoid a future of constant fire-fighting in IT.
What to do today
BMC offers a ZipKit for BladeLogic TrueSight Server Automation to automate the Heartbleed remediation process. We also have integration with Discovery / ADDM to make sure that we catch all the vulnerable systems - including ones that might have been provisioned without full IT oversight or even awareness. The whole process can also be documented in Remedy ITSM The specified item was not found..
This way, Heartbleed remediation moves from firefighting to routine background, and you can focus on activities that add value to the business.