Patch Deployment Best Practices in the Enterprise

RFG believes patch management efforts have become an expensive and time-consuming operation for most enterprises. More important, failures to patch vulnerable systems in a timely manner present significant business risk to the enterprise. IT executives should develop and document patch deployment procedures for their departments to better manage this problem. Further, to mitigate the overhead associated with this effort, IT executives should evaluate service providers offering patch identification and deployment services.

Business Imperatives:

• Unpatched vulnerabilities today constitute the overwhelming majority of security breaches in the enterprise related to viral or worm-based attacks. Unfortunately, the rate at which the developers of these attacks release exploitations for newly discovered vulnerabilities is increasing. IT executives should actively pursue patching methodologies for their organizations that allow them to quickly and effectively respond to patch requirements as they are made public.
• Patching operations can be a time-consuming and expensive proposition, especially for products from manufacturers that have displeased the developers of such exploits, such as Microsoft Corp. However, automated patching facilities can leave users unprotected or entirely without services if the patches themselves are defective. To better manage these problems, IT executives should develop patch deployment plans that establish procedures for the IT department to identify, test, and deploy patches as they are released.
• Finally, several vendors have developed services designed to assist companies in identifying missing patches and deploying them automatically. IT executives should evaluate these offerings and determine whether it would be possible to mitigate the costs associated with patching vulnerable systems without sacrificing response times to vendor alerts or thoroughness in identifying required patches.

The patching of system vulnerabilities has become one of the most expensive and time-consuming recurring administrative tasks in the enterprise. The process is also prone to failure, as viruses and worms often use unpatched vulnerabilities as the initial entry point into a protected network, and then use other techniques for propagating once inside. Thus, any of the following factors could invalidate the process:

1. Patches that are not identified and installed in time to prevent damage.
2. Vulnerable systems that were not patched when the patch was deployed.
3. Defective patches that do not properly close the vulnerability.
4. Defective patches that create new vulnerabilities, or cause loss of services.

Further, unpatched systems may present the following consequences to the enterprise as a whole:

1. Costs and overhead associated with cleanup after an infection or security breach.
2. Direct loss of revenue from system outages and productivity declines.
3. Indirect financial loss due to loss of reputation and/or customer confidence.
4. Legal liabilities from breach of sensitive records, for example.
5. Loss or corruption of business data.
6. System downtime, inability to conduct business.
7. Theft of business assets.

Clearly, this is not a task to be assigned to an entry-level technician and then forgotten. IT executives should take steps to ensure that patches are identified and deployed as quickly as possible to all vulnerable systems. The cost of doing so may be expensive, but the business risk and potential for litigation posed by a potential security breach far outweigh this cost.

Some vendors are experimenting with automated patching procedures within their products. For home users, this is an admirable goal, as it may improve security levels across the installed base of a vendor's products. However, in enterprise environments, automated, vendor-driven patching mechanisms present additional risks. In the past, some vendors have used these services to distribute "updates" that are not security-related and that alter a system's environment to the benefit of the vendor, weakening customer trust of those vendors. More importantly, defective patches can and do get released, and customers are rightfully wary of automated patching services that do not provide for the ability to regression test a patch against the target environment.

In addition, the typical enterprise has multiple versions of operating systems and applications. A patch may be version-specific, thereby requiring care before blanket deployment. One way to reduce patch complexity is development and enforcement of standard configurations. (See the RFG Research Note "Toward the Elastic Enterprise: Client IT Systems Management.")

To better manage this task, IT executives should develop detailed patch management policies that describe the processes that will be used to identify and deploy patches, and the ownership of each step in the workflow. The first step in developing such a policy is to develop a business application profile for each application in the enterprise that describes each application's sensitivity. (See the RFG Research Note "An Update on the Importance of Business Application Profiles (BAPs).") The final output of this project should be two critical factors: the business risk associated with a security breach, and the allowable downtime periods to meet patching requirements for different vulnerability risk levels. Using this information, IT executives should develop a matrix similar to Figure 1 below.

Source: Robert Frances Group

IT executives should ensure that determination of business risk includes both direct and indirect factors. Many enterprises do not consider e-mail clients to present a high business risk because alternate access methods, such as Web-based facilities, can be offered to users during downtime events. However, because e-mail clients represent one of the most common sources of exposure to infection from viruses and worms, they pose an indirect risk to other systems in the enterprise. E-mail clients should therefore be patched as soon as possible after a patch is made available.

IT executives should then develop procedures for patching systems, with clear identification of ownership for each role involved. These procedures should include notification requirements to help identify missed steps, and should also include methods for dealing with vulnerabilities for which patches have not yet been released (or never will be, as may be the case for unsupported products). Figure 2 below provides a sample list of procedures for affecting this process.

Source: Robert Frances Group

Note that at each step of the process the individuals that own that role must report the outcome of the step to another individual or group. No step can be considered complete without this notification, and IT executives should have their teams perform notifications in writing, to provide a trail of activity during patching procedures. This documentation should be incorporated into a change management process initiated if it is determined that a necessary patch was not deployed, to determine the source of the problem and update the procedures list to catch it in the future.

There is no need to flood individuals such as the CIO/CSO with documentation; summary reports that describe the final outcome of each patching operating are typically sufficient. However, because the security of the company, and thus IT executives' reputations, are on the line, at a minimum IT executives should be alerted when patching procedures are not addressing the enterprise's needs. In addition, the documentation produced from notification steps should be immediately available to those executives for review as necessary.

To identify patches, employees should monitor both vendor Web sites and security mailing lists and portals such as SecurityFocus and VulnWatch. (See the RFG Research Note "E-Commerce Application Security Best Practices.") Also, during the identification process, IT executives should be wary of allowing their teams to rely solely on the vendor's assessment of vulnerability risk levels. Although this does not occur every day, product vendors have been known to be relaxed in their classification risk levels due to political interests.

IT executives should also not allow the audit team review of patches deployed to rely on internal assessments of required patches. Rather, the group that performs this function should itself monitor vulnerability lists and vendor Web sites. By comparing their own lists of vulnerabilities to address against documented patch deployments from the security and administrator groups, the audit group forms a system of checks and balances that helps ensure patches will not be missed or improperly deployed.

Finally, IT executives should evaluate third-party patch identification and management products and services to determine whether they can effectively reduce costs and/or overhead associated with patch management processes in their organizations. Some of these vendors include ConfigureSoft, Inc., Ecora Software Inc., IT-Defense, Inc., PatchLink Corp., Shavlik Technologies, LLC, and St. Bernard Software, Inc. Products from these vendors can reduce workloads associated with identifying required patches, deploying those patches, and then auditing systems to ensure that all required patches have been installed. In many cases, the costs of such products are immediately offset by the reduction in overhead of the internal patch management processes, and the reduced possibility that a required patch will be missed.

RFG believes patch management efforts must be a core element in enterprise security strategies. To ensure that all necessary patches are deployed in a timely and cost-effective manner, IT executives should develop application risk assessments and patching procedures that include integral audit, change management, and notification steps. IT executives should also evaluate third-party products and services to determine if and where they should be deployed to reduce internal patch management overhead.

Copyright © 2003 Robert Frances Group, Inc. All rights reserved. Agenda products are published by Robert Frances Group, Inc., 22 Crescent Road, Westport, CT 06880. Telephone 203/291-6900. Facsimile 203/291-6906. http://www.rfgonline.com. This publication and all Agenda publications may not be reproduced in any form or by any electronic or mechanical means without prior written permission. The information and materials presented herein represent to the best of our knowledge true and accurate information as of date of publication. It nevertheless is being provided on an "as is" basis. Reprints are available.