Story 003 - When an OS patching broke production
Published:
The chatbot had been running reliably for months now. Then came a routine OS patching task performed by the infra team, updating the OS from RHEL 9.7 to RHEL 9.8. After the maintenance window, I noticed the chatbot was down again. I SSHed into the server and ran the nvtop command, “No GPU to monitor.” That looked suspicious so I checked if the instance type was correct using neofetch, it showed g5.xlarge. I checked my vLLM service status and it had exited with a failure. Then I checked with nvidia-smi command, it failed and told me to make sure the latest NVIDIA driver was installed and running. So I tried rebuilding the existing driver using DKMS, hoping it would simply recompile the kernel module for the new RHEL 9.8 kernel. Instead, it gave me “Error! Bad return status for module build.” That led me to conclude the installed driver wasn’t compatible with the new kernel.
A temporary kernel fallback to RHEL 9.7 was required to restore the GPU service. Downloading a newer NVIDIA driver and validating the entire CUDA stack would take time, and the priority was production restoration rather than immediately fixing the OS patching. I proposed a break-glass recovery plan: temporarily boot the server back to previous working RHEL 9.7. This wasn’t intended to bypass patch compliance, but was simply the fastest and safest way to bring the chatbot back online while I worked on a permanent fix for RHEL 9.8. After receiving approval, I rebooted the instance to the previous kernel and restarted the vLLM service. Everything was back to normal.
Now the real work started. A newer driver version was required. After some research, I decided to upgrade both NVIDIA Datacenter Driver and the CUDA Toolkit together. I went to the NVIDIA Datacenter Driver downloads page and searched for the latest driver that officially supported RHEL 9.8, and downloaded both the updated driver version and the newer CUDA Toolkit RPM packages. After copying them to the server, I removed the older driver, installed the newer driver, upgraded the CUDA Toolkit from 12.9 to 13.2, and rebooted back into the RHEL 9.8 kernel. Then came the moment of truth. nvidia-smi detected the NVIDIA A10G again, CUDA initialized successfully, and the vLLM inference service started without issues. The final step was taking a new AMI backup from the repaired instance.
Looking back, this incident taught me that production AI systems extend far beyond Python environments. A simple operating system update can break the contract between the kernel and the GPU driver long before your application ever starts. Maintaining AI infrastructure isn’t only about serving models; it’s about understanding every layer beneath them.