Patching Guide
Pre-Checks
mkdir -p /home/CMS-Linux/Patching/Prechecks; { echo "PreChecks" ; echo "Hostname:" ; cat /etc/hostname; echo "Disk Usage:" ; df -h; echo "Block Devices:" ; lsblk; echo "Block Device Attributes:" ; blkid; echo "Memory Usage:" ; free -m; echo "DNS Configuration:" ; cat /etc/resolv.conf; echo "IP Address Information:" ; ip a; echo "Mounted Filesystems:" ; cat /etc/mtab; echo "Filesystem Table:" ; cat /etc/fstab; echo "Detailed Filesystem Usage:" ; df -HT ; echo "System Time and Date:" ; timedatectl; echo "Multipath Configuration:" ; multipath -ll ; echo "IO Statistics:" ; iostat ; echo "Pacemaker/Corosync Status:" ; pcs status; echo "Chrony Service Status:" ; systemctl status chronyd ; echo "Cron Service Status:" ; systemctl status crond; echo "Kdump Service Status:" ; systemctl status kdump; echo "System Information:" ; dmidecode; echo "Enabled Unit Files:" ; systemctl list-unit-files; echo "All Services:" ; systemctl list-units --type=service --all; } > /home/CMS-Linux/Patching/Prechecks/Prechecks_$(date +'%Y-%m-%d').txt
In this part of the code, the last section says > Prechecks_$(date +'%Y-%m-%d').txt. This means that all the results from the command above will be saved into a text file named Prechecks_<current date>.txt, making it easier to locate and work with.
You can save the txt file to your home directory too by adding the path like this:
> /home/kumars63-pa/Prechecks_$(date +'%Y-%m-%d').txt
Before you start patching servers, it's a good idea to check the kernel version first. After the upgrade, compare this version to make sure the server has been updated properly and to identify any issues, like if the kernel was missed or if there are any discrepancies.
This command displays the current kernel version of the operating system.
If by any chance you notice the kernel has not been upgraded, use the below command:
Patching Process
RHEL
This command cleans the Yum cache, lists all enabled repositories, and then updates all installed packages to their latest versions
Then rebooting the server applies all the changes made during the upgrade.
Azure RHEL
This command cleans the Yum cache, fetches all available packages, and updates installed packages to address security and bug fixes. The difference between the previous command and this one is that this command specifically updates only packages related to security (
--security) and bug fixes (--bugfix).
Update OMI [Only if required]
This command updates the specific package named "omi" to its latest available version
Then rebooting the server applies all the changes made during the upgrade.
Ubuntu
This command updates the package list for available upgrades and then upgrades all installed packages to their latest versions.
Then rebooting the server applies all the changes made during the upgrade.
Post-Checks
Splunk Status
Splunk is a service used for monitoring, and analyzing logs and system performance data, in real-time.
Used to check if splunk service is running on the server.
If not, Could be restarted using above
Fetching packages
UBUNTU: This command will fetch the all the packges that were installed on the current day for artifacts.
RHEL: This command will fetch the all the packges that were installed on the current day for artifacts.
Using Multi-run
Multi-run is a script, which is used to run a command at once on multiple servers, listed in a text file (could be hostnames or IPs).
Here ./multi-run states the script file present, --byfile specifies the file it is going to refer for the server name or ip, host-name is the file (this could be named anything). --command is what initiates the command you want to run in all those servers and later within " " you specify the command you want to run.
Below are some usefull commands put together using multi-run:
./multi-run --byfile host-name --command "cat /etc/*-release | grep -i 'pretty' | awk -F= '{print $2}' | awk -F'(' '{print $1}'"
./multi-run --byfile host-name --command "OS_TYPE=\$(cat /etc/*-release | grep -i 'pretty' | awk -F= '{print \$2}' | awk -F'(' '{print \$1}' | tr -d '\"'); if [[ \$OS_TYPE =~ 'Ubuntu' ]]; then grep -E 'install|upgrade|remove' /var/log/dpkg.log | grep -Ev 'half|not'; elif [[ \$OS_TYPE =~ 'Red Hat' || \$OS_TYPE =~ 'CentOS' || \$OS_TYPE =~ 'RHEL' ]]; then yum history info \$(yum history | awk '/^[0-9]/ {print \$1; exit}'); else echo 'Unsupported OS'; fi" > Artifacts_$(date +'%Y-%m-%d').txt
Others
/boot filesystem cleanup
1. RHEL
2. Ubuntu