GPU Cold Starts: Why AI Generation Waits, and How to Show It Honestly (2026)
Scale-to-zero GPUs make some requests wait for a worker to boot. Here is what a cold start is made of, what shortens it, and why the label on the wait matters as much as the wait.
“Look scared then turn and run away with big dramatic strides”
- A cold start is the wait while a scale-to-zero GPU worker boots, pulls its container and loads the model before the first request can run.
- Scale-to-zero is the right economics for spiky workloads like animation generation: you pay for GPU seconds used, not for cards sitting idle.
- Baking model weights into the container image shortens the wait the most, because a cold start becomes a container pull rather than a multi-gigabyte download.
- What is left is a product problem. Queued, starting and generating are three different states, and showing them as one spinner makes a working product look broken.
- Warming a worker when the user focuses the prompt box, rather than when they press submit, often hides the wait entirely.
A cold start is the extra wait on a request that arrives when no GPU worker is running. Someone has to pay for that wait, and the only real choice is whether it is paid in money (keeping cards warm around the clock) or in seconds (booting one on demand).
For most AI products the seconds are the right trade. The engineering to shorten them is real but small. The bigger lever is what the user sees while it happens.
What is a cold start made of?
It is not one wait but several, and each has a different fix:
| Stage | What's happening | What shortens it |
|---|---|---|
| Queue | Waiting for a free worker | More workers allowed to run at once |
| Container pull | Downloading the image onto the machine | A smaller image, cached layers |
| Model download | Fetching weights at boot | Bake the weights into the image |
| Model load | Moving weights into GPU memory | Keeping a worker warm |
| Inference | The actual work | Not a cold-start cost |
Baking the weights into the image is the first thing worth doing, because it shortens the real wait rather than describing it. A cold start stops being a multi-gigabyte download from a model hub and becomes a container pull from a registry the provider already caches.
Why not keep a GPU warm all the time?
Because generation traffic is spiky. A card that is warm around the clock is paid for around the clock, including the hours nobody is generating anything. Scale-to-zero turns that into a cost per second of actual work, which is what lets a free tier exist at all.
The middle ground is to keep a minimum of one worker warm during your busiest hours and let it scale to zero overnight. Whether that pays for itself depends entirely on how often a cold start actually happens, so measure it before you buy it.
What should the user see during a cold start?
Nielsen Norman Group’s long-standing guidance is that about ten seconds is the limit for keeping someone’s attention on a task, and that anything longer needs progress feedback that says what is happening. A cold start blows through ten seconds easily.
A user who waits twenty seconds at an unlabelled spinner concludes the product is broken. The same user waiting the same twenty seconds next to "Starting GPU" concludes it is working. The wait is identical; the label is not.
| State | Unhelpful label | Honest label |
|---|---|---|
| Queued | Generating… | Queued, with your position |
| Starting | Generating… | Starting GPU (first run in a while) |
| Generating | Generating… | Generating, with real progress |
| Failed | Something went wrong | Failed, credits returned |
Before NPCMotion split these apart, a cold start looked identical to a hang, and it was the single most common support message. They now arrive as three separate states pushed over a channel, as recorded in the 0.7.0 changelog.
How do you hide the wait completely?
Start warming on intent, not on submit. Focusing the prompt box is a strong enough signal that a generation is coming. By the time a prompt has been typed and sent, the worker is often already up, and the cold start never shows at all.
- Pre-warm on the first meaningful signal (focus, a file picked), not on page load, which wakes workers for visitors who never generate.
- Debounce it, so a user tabbing in and out does not boot five workers.
- Let the warm-up expire on its own. A wasted pre-warm should cost seconds, never a worker idling for an hour.
Sources and further reading (checked Oct 2, 2026)
- Nielsen Norman Group, Response Times: The 3 Important Limits: nngroup.com
- Nielsen Norman Group, Progress Indicators Make a Slow System Less Insufferable: nngroup.com