Batching

Plain English. Serving many users' requests through the same chips at the same time. One person's request leaves most of an accelerator idle; stacking dozens of requests together fills it. Batching is the unglamorous reason hosted inference is cheap.

Why it moves money. Utilisation is the margin engine of the API business. One widely shared argument puts hosted serving at roughly thirty times more resource-efficient than the same model running for a single user at home — a reported estimate, but the direction is not in dispute. Batching is also why capacity has a time-of-day price: off-peak tokens are cheaper because the batches are emptier.

What to watch. Off-peak and batch-tier pricing spreading across providers — the clearest public evidence that utilisation, not capability, is the binding constraint.

From the signals. The local-models argument is really an argument about batching economics: hosted serving's claimed ~30x resource-efficiency edge. Anthropic's capacity constraints produced "batch AI" and timezone arbitrage — off-peak allocation as policy.