Understanding Patching on Linux



WHAT IS A PATCH?
A patch is a piece of software code that will be inserted into the existing program in the system. This is an imidate fix to existing software before a minor release is planned. A patch is a kind of temporary and quick fix to existing software.

WHAT IS A PATCH LOOKS LIKE?
Suppose you have some xyz.c file within your Linux kernel which requires a patch, the patch is just the difference between the existing line of code in that file and extra lines which we will add to this file. It is just a diff of lines of code which is added to an existing file.

Enterprise-level Approach

On Red-Hat based systems we use the yum package manager as the preferred method to install and update packages.



Patching is generally motivated by system security. Remember that IT security, in general, has three aspects:

  1. Confidentiality - wrong people must not get access to certain things (at least password files and other authentication infrastructure; possibly entire systems and the information in them).
  2. Integrity - the information processed must not be damaged in the process, neither by bugs nor by malice.
  3. Availability - the systems and the information in them must be available for use by the right people; that's why they exist in the first place.

Another stage in the patching we do is a yum clean all and yum repo list so that when new patches are published through a content view, the cached information doesn't "get in the way" of a new set of patches, while not the rule, there are enough "exceptions" where this is an issue enough to warrant doing this in that environment. 

The important point to understand in Patching

Patching helps in keeping the machines in an enterprise updated with security fixes and critical bug fixes, especially in a data center or a server farm.

  • Set up Linux RPM Repository based in Unbreakable Linux Network (ULN) channels.
  • Download Advisories (Erratas) from ULN.
  • Set up Linux Patching Group to update a group of Linux hosts and collect compliance information.
  • Allow non-compliant packages to be patched.
  • Rollback/Uninstall packages from the host.
  • Manage RPM repositories and channels (clone channels, copy packages from one channel into another, delete channels).
  • Add RPMs to custom channels.
  • Manage Configuration file channels (create/delete channels, upload files, copy files from one channel into another)

Patching Best Practices

The SysAdmin, Audit, Network, and Security (SANS) organization lays out best practices for patch management. These best practices give administrators guidance on how to implement a corporate policy that documents, audits, and assesses risk across the organization to determine when and how patches should be deployed. 

Best practices are:

  • Know your environment. Before you know what needs to be patched, you need an audited list of all Linux systems on the network.
  • Assign risk levels to each server. Risk levels tell administrators which servers are most important and should be prioritized. All systems should be patched, but targeting the most important servers will lower the risk of them being compromised while testing and other patching is underway.
  • Consolidate patch management software into one solution. Automation tools are beneficial, but too many different tools making changes to the environment can lead to errors and possible race conditions. 
  • Review vendor patch announcements regularly. Automation tools will download updates automatically, but administrators should still be aware when new patches are available, especially critical ones.
  • Mitigate risks of patch failures. It’s not uncommon for administrators to halt updates due to an issue with exceptions. When this happens, servers should be locked down to avoid exploit potential.
  • Always test patches in staging first. A staging environment should replicate production, so patches can be tested and lower the risk of downtime.
  • Patch systems as quickly as possible. The longer a server is unpatched, the bigger the risk of compromise due to a known vulnerability.
  • Use automation tools. Automation tools take a lot of the overhead from administrators and automatically deploy patches when they are available.


How Often Should Patching Be Performed?

Installing anything on a production server should be done after thorough testing. In a large enterprise environment, it’s possible for new updates to be available every day, which means constant testing and deployment. Manually checking for new patches every day is tedious and requires unnecessary overhead when patch management automation is available.

To add to the overhead, patches should be deployed only after being tested in a staging environment. Staging environments should be a replica of production to ensure that it’s a 1:1 match during testing or errors could cause downtime in production. Even though testing is important, a good rule of thumb is to apply patches within 30 days of vendors making them available. 

For security patches, it’s critical that administrators test and deploy them as soon as possible. Zero-day vulnerabilities are a real threat to organizations and their digital assets. When zero-day vulnerabilities are announced, threat actors quickly create exploits to take advantage of unpatched systems. Several recent data breaches were the result of exploits on unpatched systems. To lower the risk of a data breach, organizations should rapidly deploy security patches as soon as they are available.

What is Linux Patch Management Strategies?

Unlike closed-source operating systems like Windows, Linux patching can be a bit more unpredictable and complex. Open-source has its advantages, but one disadvantage is running an operating system with several possible changes made by various contributors. Just one incompatible change could affect your entire organization.

To alleviate some of the overhead and hassles of poor patch management, here are a few strategies and best practices to incorporate into your procedures:

  1. Create a patch management policy. This policy should include every step including quality assurance (QA) testing, frequency of patching, any rollback procedures, and who must sign off on operating system changes.
  2. Use scanning tools that find vulnerabilities. Whether it’s public-facing servers or internal hosts running corporate applications, vulnerability scans will find unpatched systems to avoid common exploits.
  3. Use reporting to identify failed patches. How do you know that a patch installation was successful? A good patch management solution has a central dashboard that displays reports on successful and failed patch installations so administrators can review and manually patch the system if necessary.
  4. Deploy patches as soon as testing is complete. Testing is important before deployment, but as soon as testing provides a green light for deployment, patches should be installed across the entire environment.
  5. Document changes to the environment. Usually, documentation is done in the form of change control where authorized employees sign off on updates to the environment. This step is important when reviewing downtime and performing root cause analysis. It’s also important for auditing and compliance reasons.

How to Patch Your Linux Systems Manually?

Even with patch automation, manual updates are occasionally necessary. After a failed update, administrators may need to manually patch the system. Manual updates might be necessary in a testing environment. The commands to update Linux depend on your distribution, but here are the commands for some common distributions.

For Debian-based distributions (e.g. Debian, Ubuntu, Mint), the following commands will let you view available patches and update packages and the operating system:

sudo apt-get update

sudo apt-get upgrade 

sudo apt-get dist-upgrade

 

For Red Hat Linux distributions (e.g. RedHat, CentOS, Oracle), the following commands check for updates and patches the system:

yum check-update

yum update

Roll Back a patch in RHEL/OEL/Centos

Step 1: Boot from the old kernel.
[root@funebs122 ~]#uptime

Step 2: Now run the yum history command to view a history of previous yum activities.

[root@funebs122 ~]# yum history 

View more information about this transaction by typing yum history info <id>

[root@funebs122 ~]# yum history info 7

Step 3: Now, we will roll back this transaction via the yum history undo command.

yum history undo 7

Step 4: Reboot your system


If you like please follow and comment