Why Napkin Math Still Matters in the Age of AI
In an industry that moves at the speed of GPU releases and model version announcements, it can be tempting to defer all cost analysis to a spreadsheet, a cloud pricing calculator, or a vendor's sales team. But some of the most clarifying insights in AI infrastructure come from rougher, faster methods — the kind you can sketch on the back of a napkin. When it comes to AI inference cost at scale, napkin math isn't a shortcut. It's often the most honest tool available.
This article breaks down how to think about inference costs using first-principles estimation, why those estimates matter more as you scale, and what the numbers tend to reveal about architectural trade-offs that are easy to miss when you're buried in dashboards.
What Is AI Inference, and Why Does It Cost What It Does?
Inference is the process of running a trained machine learning model to produce an output — a completion, a classification, a summary, an embedding. Unlike training, which is a one-time (or periodic) cost, inference is ongoing. Every user query, every API call, every automated pipeline trigger generates an inference event. At scale, this becomes the dominant line item in your AI budget.
The cost of inference is driven by a relatively small set of variables: the number of parameters in the model, the number of tokens processed (both input and output), the hardware running the model, and the utilization rate of that hardware. Understanding how these variables interact is where napkin math becomes indispensable.
The Core Variables: A Napkin Math Framework
Model Size and Memory Footprint
A rough rule of thumb in the industry is that each billion parameters in a model requires approximately 2 GB of GPU memory when loaded in FP16 (half precision). A 70-billion-parameter model therefore needs around 140 GB of VRAM just to load — before you account for KV cache, activations, and batch overhead. This immediately constrains which hardware configurations are viable and pushes you toward multi-GPU or multi-node setups.
Smaller models — in the 7B to 13B range — can often fit on a single high-end GPU, dramatically reducing infrastructure complexity and cost. This is why model size selection is not just a quality decision; it is a cost architecture decision.
Token Throughput and Latency Trade-offs
Inference pricing from major providers is typically expressed in dollars per million tokens. At first glance, even premium model pricing looks affordable — a few dollars per million input tokens, slightly more for output tokens. But scale changes that perception quickly. An application generating 100 million tokens per day is spending thousands of dollars daily on inference alone, before factoring in any surrounding infrastructure.
Latency requirements compound this further. High-throughput batch inference can be run efficiently with large batch sizes, improving GPU utilization and lowering per-token cost. Real-time conversational inference, however, demands low latency, which typically means smaller batches, more idle GPU time, and a higher effective cost per token. These two workloads are almost inverse in their optimization profiles.
GPU Hours and Hardware Utilization
A modern H100 GPU can be leased from major cloud providers for roughly $3 to $5 per hour depending on region, reservation type, and provider. If your model requires four H100s to serve at reasonable throughput, your raw hardware cost is $12 to $20 per hour — or roughly $288 to $480 per day per inference cluster. A napkin calculation comparing this against your expected token volume quickly tells you whether self-hosting is financially competitive with API-based inference at your specific scale.
The critical variable here is utilization. A GPU cluster sitting at 30% utilization is effectively tripling your per-token cost compared to one running at 90%. Optimizing for utilization — through request batching, traffic shaping, and model serving frameworks like vLLM or TensorRT-LLM — is often where the most significant cost reductions come from in practice.
What Scale Changes About the Math
Below a certain threshold of usage, the managed API approach almost always wins on cost efficiency. You pay only for what you use, you avoid capital expenditure, and you benefit from the provider's optimization work. The napkin math tips in the other direction — toward self-hosting or dedicated deployments — somewhere in the range of tens of millions to hundreds of millions of tokens per day, depending on the model and hardware generation.
Beyond raw token volume, scale also surfaces hidden costs that are easy to underestimate in early-stage deployments. These include observability and logging infrastructure, model versioning and rollback capabilities, safety and content filtering layers, and the engineering time required to maintain a production inference stack. None of these show up in a simple tokens-per-dollar calculation, but all of them appear on the eventual invoice.
Practical Implications for Architecture Decisions
Right-size your models. The most capable model is rarely the most cost-efficient model for a given task. Evaluating smaller, distilled, or fine-tuned models against your specific use case can yield dramatic cost reductions with minimal quality degradation.
Separate your workloads. Batch and async workloads should be architected differently from real-time inference. Running everything through the same pipeline is a common source of unnecessary cost at scale.
Track tokens, not just API calls. Cost visibility at the token level, broken down by model, use case, and user segment, is essential for identifying optimization opportunities before they become budget crises.
Build for utilization from day one. Request queuing, dynamic batching, and load balancing are not premature optimizations — they are the foundations of cost-efficient inference infrastructure.
The Value of Rough Numbers Done Early
The appeal of napkin math is not precision — it's speed and clarity. A back-of-envelope calculation completed in ten minutes often surfaces the same structural conclusions as a detailed financial model that takes a week to build. In AI infrastructure planning, where the cost landscape shifts with every new model release and hardware generation, the ability to quickly reframe assumptions and rerun estimates is more valuable than false precision.
Understanding inference cost at scale is ultimately about developing intuition: knowing which variables dominate the calculation, recognizing when a workload is crossing a threshold that changes the optimal architecture, and being able to communicate trade-offs clearly to stakeholders who may not share your technical context. Napkin math, done well, builds exactly that kind of intuition — and in a domain where the numbers can spiral quickly, that intuition is worth developing early.