Story 002 - The Curious Case of Missing G5 Capacity

Every morning the chatbot automatically started and every night it shut down to save costs. Then one morning… the scheduler failed. Not because of code. Not because of CUDA or vLLM. AWS simply said “Insufficient capacity”. Both the preferred instance families (g5.xlarge and g6.xlarge) were unavailable in the ap-south-1 region. Even attempts to resize to larger variants still resulted in insufficient capacity. That was the first time I learned that cloud infrastructure isn’t infinite.

Waiting wasn’t an option. So I started investigating alternatives. Eventually, I proposed a temporary instance type change to g4dn.12xlarge. At first glance, it looked like an expensive upgrade instead of handling the insufficient capacity. Four NVIDIA T4 GPUs, 64 GB of aggregate VRAM. Surely 4 GPUs should outperform one A10G, handling the outage at least. Except…engineering isn’t about adding numbers. It’s about understanding architecture.

More GPUs, more problems. My first instinct was to preserve the existing deployment. Instead of changing the application, I tried changing how it used the hardware. I configured vLLM to serve the LLM using –tensor-parallel-size 3, as the T4 GPUs only provided 16GB of VRAM each, the model would be sharded across multiple devices. While the embedding and reranking models occupied the fourth GPU. On paper, this deployment looked reasonable. In practice, it exposed the architectural differences between the two instance families. With Tensor Parallelism, GPU-to-GPU synchronization became part of every inference request. As the conversations grew longer, synchronization overhead, BF16 compatibility limitations, and lack of efficient peer-to-peer communication caused the inference server to become increasingly unstable. While the model math suggested sufficient room, the overhead of managing KV Cache for ~20k context length across shared cards proved to be significantly higher than on a single card. The system encountered Silent OOMs where the process was killed by the OS without a log trace as the peak spikes during the 2nd message exceeded the T4’s hardware-level buffers. That was the moment I realized something important. More GPUs didn’t necessarily mean a better deployment.

Preserving the service, not the model. Instead of forcing the original model to fit the hardware, I changed what i was trying to preserve. The solution was surprisingly simple. Instead of an 8B model distributed across multiple GPUs, I switched to a 3B parameter model that fit comfortably with the required max-seq-length on a single 16GB T4. The fallback deployment became: GPU0 - smaller LLM, GPU1 - Embedder+Reranker, and yes half of the available GPUs sat idle. But the chatbot became stable again. The only compromise was the generator’s parameter size. The retrieval pipeline remained unchanged. Operationally, the fallback became effortless. Whenever AWS capacity forced us onto the g4dn.12xlarge, all that was required was stopping the primary vLLM service and replacing it with the fallback vLLM service configured for the smaller model. No application changes, just a service switch.

Beyond the immediate fix, the fallback deployment solved the production problem, but it left me with another question. Two of the four T4s were sitting completely idle. That didn’t sit well with me. When I later gained access to AWS Bedrock for subsequent projects, I revisited the problem from a different angle. Instead of asking how to better utilize the T4 GPUs, I asked a different question “During outages, can I remove the LLM from the GPU altogether while preserving the retrieval pipeline?”

That led me down another rabbit hole. I started exploring compute-optimized instance types to move the cross-encoders. I found the C8i instance types, after digging deeper I found the Intel Xeon 6 architecture, AMX acceleration, and OpenVINO. The idea was to migrate the cross-encoders to the c8i.4xlarge instance using Intel AMX acceleration, while offloading the generator LLM to Amazon Bedrock. Once again the retrieval pipeline would remain constant. I think this was surprisingly elegant. Operationally, the transition would again simply become a systemd service switch, given the AMI would be pre-configured to allow both inference stacks and their dependencies to coexist. I never implemented this architecture. Not because it wouldn’t work, but because justifying the engineering effort and operational changes inside a production environment wasn’t worth the organizational overhead at that time.

Looking back, I realized the outage had taught me to design systems that degrade gracefully. I stopped thinking in terms of GPU servers and started thinking in terms of inference architectures. The best production system isn’t the one that only works under ideal conditions. It’s the one that keeps serving users even when the infrastructure changes underneath it.