Story 001 - First production deployment (feat. RHEL EC2)

After proving the chatbot locally, the next step was obviously to move inference to a cloud GPU infrastructure for user use. Unlike Story 000, where I had to justify the workstation hardware, this time I had to justify cloud hardware, which was even more exciting.

I spent time comparing AWS GPU instances before making a recommendation. I needed a balance between inference performance, VRAM, and cost. There were two shortlisted options: the g6.xlarge and g5.xlarge. I eventually settled on a g5.xlarge, powered by an NVIDIA A10G GPU, because it had 600GB/s memory bandwidth compared to 300GB/s of g6’s L4 GPU. For my LLM inference workload, I expected the higher memory bandwidth to provide better inference throughput. This decision was also driven by the fact that the NVIDIA Ampere architecture would be consistent across the development and production environments.

The instance was provisioned with RHEL OS. A few emails later, I received something every engineer loves to see: “You have been provided sudo privileges.” I immediately SSH’d into the server and ran the nvidia-smi command. I expected to see a GPU waiting for me. But to my disappointment, nothing… no GPU. After a Google search, I knew the GPU driver was missing, due to which the operating system had no idea a powerful NVIDIA A10G was attached to it. That was my first lesson in infrastructure. Provisioning a GPU instance doesn’t automatically create an AI-ready machine.

So I headed to NVIDIA’s datacenter driver download page, Selected Product: NVIDIA A10G, Operating System: RHEL, Architecture: x86_64. Downloaded the datacenter driver, copied it to the EC2 instance using scp, installed it, and rebooted the machine. Then I ran the nvidia-smi command again. This time, the A10G appeared.

Of course, seeing the GPU wasn’t the end. It was only the beginning. Now I had to do the actual work. Install the CUDA Toolkit from NVIDIA’s RHEL RPM package, create a Python venv, install torch+cu and vLLM, download the model weights, clone the code repo, and install the requirements.txt. Every layer from the OS to the application had to be built before the first token could ever be generated. Eventually torch.cuda.is_available() printed True.

Only then did I deploy the chatbot application, bind Flask to 0.0.0.0, and hand the server over to the infrastructure team as the EC2 was placed behind an ALB, which needed the DNS mapping to a CNAME, and forwarded HTTPS traffic on port 443 to the Flask application’s listening port. Looking back, I thought I was deploying an AI application. In reality, I was learning something much more fundamental. Cloud providers provision infrastructure. Engineers provision platforms.