Before You Upgrade Hardware, Fix the Software

Better software algorithms can significantly improve effective memory efficiency, but only until the workload reaches a real hardware bottleneck. The Misconception Recent work such as Google's TurboQuant shows that software can significantly reduce memory pressure for specific workloads like LLM inference. At the same time, companies across the AI stack are investing in physical infrastructure such as power and chips to sustain growing compute demand. Meta has expanded its energy strategy, including major nuclear power agreements for AI-related infrastructure, while NVIDIA remains tied to the semiconductor path through advanced chip production and packaging. Together, these trends raise a broader question: if software can make systems more efficient, how often are we upgrading hardware before we have truly exhausted software optimization? The Impact The decision to optimize software or upgrade infrastructure is not only a technical choice. It affects cost, scalability, engineering time