Story 003 - When an OS patching broke production
Published:
The chatbot had been running reliably for months now. Then came a routine OS patching task performed by the infra team, updating the OS from RHEL 9.7 to RHEL 9.8. After the maintenance window, I noticed the chatbot was down again. I SSHed into the server and ran the nvtop command, “No GPU to monitor.” That looked suspicious so I checked if the instance type was correct using neofetch, it showed g5.xlarge. I checked my vllm service status and it had exited with a failure. Then I checked with nvidia-smi command, it failed and told me to make sure the latest NVIDIA driver was installed and running. So I tried rebuilding the existing driver using DKMS, hoping it would simply recompile the kernel module for the new RHEL 9.8 kernel. Instead, it gave me “Error! Bad return status for module build.” That led me to conclude the installed driver wasn’t compatible with the new kernel.