Every company has to update and patch its software, but unless the process is carefully managed, serious problems can occur. How can you make sure you’re following the right steps?
To ensure a predictable rollout when a patch is deployed across your network, it is important to test it first in a nonproduction environment. Companies install software and firmware patches to fix bugs, remove vulnerabilities and add new features, but without prior testing they can put systems at risk or make them unstable.
Employing the following security patch testing best practices will reveal any conflicts with existing configurations unique to the systems in which the patch will be installed, thus avoiding interruptions to production operations.
Determine what’s most critical
Identifying which security issues and software updates are relevant to your environment requires a strong asset and software inventory. In complex environments, the best way to maintain this process is to use an automated patch monitoring and management tool.
Assign all assets a criticality level to help assess business process importance, optimum downtime periods and vulnerability risk levels. Prioritize patches, with critical patches tested, scheduled and deployed before those that are less essential. Change management is vital. The change management system must perform and track every update and each step of the patch management process. The change management policy should describe the processes used to identify and deploy patches, as well as the ownership of each step in the workflow.
Vendors routinely issue patches. Making sure you’re aware of critical patches is important. Another helpful tool is the Common Vulnerability Scoring System, which rates the severity of security vulnerabilities across a wide range of software products. Because it is application- and vendor-neutral, security teams can more easily gauge the impact of vulnerabilities on their systems and prioritize which vulnerabilities to fix first. For internally developed applications, use a software composition analysis tool. These track all the open source and third-party components used by an application and are the best way to keep track of relevant vendor update and patch announcements.
Once a patch is downloaded, verify its source and integrity. A digital signature is typically provided for this purpose. Unless there is an imminent threat, take time to see what the relevant forums are saying about the patch and determine if anyone is reporting problems post-installation. When following security patch testing best practices, it’s not always best to be first.
Replicate environments through virtualization
Virtualization is a valuable part of a patch testing strategy because you can replicate various production environments on one computer, preferably using the same hardware. Running several OSes virtually saves time, money and space. This lets you verify that applying patches will not result in unexpected or undesirable system behavior. That said, infrastructures have become far more complex and exposing a patch to as many scenarios and state spaces as possible is difficult. Cloud services, such as Azure and AWS, are a cost-effective way to create a dedicated patch testing environment that’s identical to your production system.
Patch testing is more than ensuring devices reboot correctly. Check the patch has been deployed successfully, and run smoke tests to determine if all the main functions of the software appear to work correctly. Look closely for unanticipated changes within the test environment, such as the following:
- Program failures
- Changes in permissions
- Newly disabled services
- Newly enabled services
- Disrupted services
- Negatively affected code
- Any other application failures
If testing produces an unsatisfactory result, identify the root cause of the problem before going any further.
Production rollouts — if conducted in stages — is another part of testing. Apply the initial rollout to less critical systems; if they perform as expected, the rollout can continue until all systems are updated. The testing process is finished when the full rollout is complete and no issues are reported within a week. If any legacy system cannot be patched, it’s vital to devise an alternative strategy to mitigate any resultant threats.
Be careful when rebooting
At least 90% of patch deployments will require a system reboot. Plan a maintenance window to apply the patches to the production system to eliminate an unexpected reboot that could interfere with business operations or cause any other problems. Consider a variety of patch strategies, such as conducting updates on a regular monthly schedule, a big weekend event to patch all systems at once or even deploying small updates throughout the month. The goal with security patch testing best practices is to avoid unexpected patching problems appearing all at once.
Even with a thorough testing program, it is wise to have a contingency and rollback plan in case something goes wrong so systems can be restored to their prepatched state. Make a backup or image snapshot of all systems before starting patch deployment. If staffing is an issue, consider an outside vendor. There are companies that provide patching services and test patches for common third-party applications and prepare scripts for deployment to production systems. Be wary, however, of using auto-update tools on mission-critical systems because you will have far less control over when patches are applied.
Patching plays a key role to ensure systems remain secure. Various audits, standards and regulations require patch deployment reports. Fines and post-breach costs can be extremely high if it’s found that an organization’s systems and client data were exposed due to running unpatched software.
Source: SEARCHSECURITY