Skip to main content
Originally published August 24, 2025 · Updated September 6, 2026 Lazy loading lets a container start before every byte of its image has been downloaded. It can shorten the wait before the application process begins. It does not eliminate data transfer, model initialization, or the time needed to produce the first useful inference response. For an inference endpoint, this distinction matters. A fast container start can move work into the first request instead of removing it. Measure readiness and user-visible response latency alongside container startup.

​
What changes when an image is loaded lazily?

With a conventional full image pull, the container runtime obtains and unpacks the required image layers before starting the application. A lazy-loading snapshotter can provide a filesystem whose data is fetched on demand. Files appear available to the application, but reading uncached data can require a remote fetch. The containerd Stargz Snapshotter project documents this behavior and the resulting tradeoff: faster image availability can introduce delays while the application reads files. Prefetching likely-needed content and caching it can reduce those delays. The benefit depends on access patterns and network conditions. Nydus RAFS is another design that separates filesystem metadata from content to support lazy access. RAFS and eStargz have different formats and implementation details. A filesystem index, compressed chunk layout, and runtime fetch mechanism must be described for the particular snapshotter being evaluated; details from one design should not be assumed to apply to another. These upstream designs explain the mechanism. They do not establish a measured Tensorfuse speedup or identify the exact snapshotter version running in a particular cluster.

​
Follow the complete startup path

These phases can overlap. Do not add individually measured durations unless their start and end boundaries are non-overlapping. Python imports can touch many small files. Model weights can require substantial reads even when the rest of the image is never used. Engine initialization and compilation can also remain after image work becomes faster. The critical path is determined by the files and compute the application actually needs.

​
Why can the first request still be slow?

A readiness endpoint may not exercise every path used by real inference. A later request may load additional assets, trigger deferred initialization, or encounter an uncached filesystem read. This does not imply that every model server behaves that way; it is a reason to measure the request path explicitly. Make the readiness condition meaningful for the application. Kubernetes readiness probes determine whether a container is ready for service traffic. A process being alive is a different condition. Check that the probe calls the intended endpoint on the correct port, then send a representative request and verify its output.

​
Measure the benefit without moving the goalposts

Compare the same image contents, model, machine type, engine version, and request workload. Record the registry region, network conditions, snapshotter version, prefetch policy, and cache state. Measure several restarts rather than selecting the fastest run. Report time to process start, time to readiness, first-request TTFT, and subsequent request latency. Include failures and bytes transferred. Keep empty-cache and warm-cache runs separate; combining them can conceal the behavior of a newly provisioned node. This revision removes the earlier context-free speedup table. It presents an evaluation method, not a replacement benchmark. For the next steps, use the vLLM cold-start guide, serving benchmark guide, and scale-to-zero cost calculation.