Story 002 - The Curious Case of Missing G5 Capacity
Published:
Every morning the chatbot automatically started and enery night it shut down to save cost. Then one morning… the scheduler failed. Not because of code. Not because of CUDA or vLLM. AWS simply said “Insufficient capacity”. Both the preferred instance families-g5.xlarge and g6.xlarge-were unavailable in the ap-south-1 region. Even attempts to resize to larger variants still resulted in insufficient capacity. That was the first time I learned that cloud infrastructure isn’t infinite.
Waiting wasn’t an option. So i started investigating alternatives. Eventually i proposed a temporary instance type change to g4dn.12xlarge. At first glance, it looked like an expensive upgrade instead of handeling of insufficient capacity. Four NVIDIA T4 GPUs, 64 GB of aggregrate VRAM. Surely 4 GPUs should outperform one A10G, handeling the outage at the least. Except…engineering isn’t about adding numbers. It’s about understanding architecture.
More GPUs, more problems. My first instinct was to preserve the existing deployment. Instead of changing the application, I tried changing how it used the hardware. I configured vLLM to serve the LLM using –tensor-parallel-size 3. While the embedding and reranking models occupied the fourth GPU. On paper, this deployment looked reasonable. In practice, it exposed the architectural differences between the two instance families.