OCI Model Artifacts: Faster Kubernetes Model Cold Start

The number that made me read the whole paper was not the 208x. It was the 40.7 minutes. An autoscaler that needs forty minutes to add a replica of a 70B-class model is not scaling, it is announcing a decision to scale some time after the traffic has gone. Most of that time is not the GPU or the engine. It is the weights travelling to the node again.

This post is a close reading of arXiv:2607.16596 (Georgii Kliukovkin, IEEE Access 2026), the KServe pull requests it describes, the Kubernetes image volume documentation and the OpenSSF model-signing project. I did not run the benchmark myself. Every figure below is from the paper or its harness, and I say where the paper's own limits cap what you can conclude.

The measured gap: warm start is flat, cold start is not

The paper measures time to first prediction for fp16-sized artifacts of 2, 14 and 140 GB, matching 1B, 7B and 70B models, on a single AWS m6i.2xlarge running a kind cluster with Kubernetes v1.36.1 and containerd 2.3.1. The registry and the MinIO S3 endpoint sit in the same cluster, so there is no WAN noise. Numbers are medians of 10 runs, 5 for the 140 GB tier. Warm means the image is already in the node's containerd store; for S3 there is no node cache, so warm just means downloading again.

Read the table by column. At 2 GB nothing matters much: oci+native:// cold (38.6 s) is even a second ahead of S3 cold (39.6 s), because fixed per-pod costs dominate. At 140 GB the warm OCI row stays at 11.7 s while S3 needs 2,441.6 s, about 40.7 minutes, which is the paper's 208x. Modelcar is 155x faster warm and carries roughly 4 s of sidecar lifecycle overhead over the image volume. Warm delivery is flat in model size because the kubelet only has to mount something already on disk.

Three ways to put weights in a pod

KServe now has more than one OCI path, and the differences are about who pulls the bytes. In modelcar the model image runs as a sidecar sharing the pod filesystem and the container runtime pulls it. In oci+native:// the webhook rewrites storageUri into a Kubernetes image volume (KEP-4639), so the kubelet pulls it and mounts it read-only with no init container and no sidecar. In oci+fetch:// the storage initializer pulls the layers itself and extracts only the models directory. The object storage path downloads into a shared emptyDir every time.

The Kubernetes side is small. An image volume is stable from v1.36, was first available in v1.31, and the documentation says it is mounted read-only in a single directory and resolved at pod startup according to pullPolicy. The next block is that shape applied to a model, with the subPath that modelcar-compatible layouts need. The image reference is a placeholder.

# Shape from the Kubernetes image-volumes docs, applied to a model.
# The reference is a placeholder. Pin by digest, not a mutable tag.
apiVersion: v1
kind: Pod
metadata:
  name: model-server
spec:
  containers:
  - name: server
    image: registry.example.com/serving/runtime:1.0
    volumeMounts:
    - name: model
      mountPath: /mnt/models
      subPath: models          # subPath on image volumes needs Kubernetes 1.33+
  volumes:
  - name: model
    image:
      reference: registry.example.com/models/llm-70b-fp16@sha256:<digest>
      # IfNotPresent: a cached node mounts instantly. Never fails the pod
      # outright when any layer is missing locally, so do not use it on
      # nodes that have not been warmed.
      pullPolicy: IfNotPresent

Why the first pull costs up to twice a download

Cold OCI starts are the slow cell in the table. At 140 GB both OCI paths took about 4,585 to 4,603 s against 2,318.8 s for S3, roughly 2x. At 14 GB the ratio is 2.6 to 3.2x. The paper blames containerd: a layer pull writes the blob into the content store and then unpacks it into a snapshot, two passes over the same disk, while the storage initializer streams an object into the pod volume in one pass. Kubelet-reported pull durations account for essentially the whole gap.

Two honest caveats come from the paper itself. The 14 GB S3 numbers are flattered by MinIO's page cache, which disappears at 140 GB because that is four times the machine's RAM. And absolute times are bounded by a 125 MiB/s gp3 volume shared by every path. The authors list zstd, lazy pulling (eStargz, SOCI) and multi-layer parallelism as possible fixes for the cold penalty but did not test any of them, and neither will I pretend they work.

The trade is therefore a one-off cost for a recurring saving. A cold OCI start also leaves roughly the artifact size behind in the containerd store, within 5 percent at 14 and 140 GB, and that retained data is the warm cache. S3 keeps no node state but makes a full transient copy on every start.

Packaging and wiring it in KServe

A model image is a plain image whose models directory holds the weights. The paper's artifacts were single-layer images with that layout, and a Busybox base is the pattern one KServe vendor guide recommends, including splitting large safetensors files across layers because some registries cap layer size. The paper only tested single-layer artifacts, so multi-layer pull behaviour at 140 GB is unmeasured.

# Containerfile: weights under /models, nothing else.
FROM busybox
RUN mkdir -p /models && chmod 775 /models

# Some registries cap layer size. Split big safetensors across layers,
# two files per layer, and put config files last. (The paper only tested
# single-layer artifacts, so measure multi-layer pulls yourself.)
COPY models/ /models/

Turning it on is a cluster flag plus a scheme. enableOciModelSupport is an opt-in that defaults to false and, set alone, gives modelcar behaviour. ociModelMode picks the cluster-wide default between modelcar and native for plain oci:// URIs, and an explicit oci+native:// or oci+fetch:// scheme overrides it for one service. The older enableModelcar remains as an alias. The webhook also sets an advisory OciImageVolumeCompatible condition on clusters that cannot do image volumes; it does not affect Ready.

# inferenceservice-config (ConfigMap), storage initializer settings.
# Opt-in master switch, then the cluster-wide default for plain oci:// URIs.
enableOciModelSupport: true      # default false; alone it behaves as modelcar
ociModelMode: "native"           # "modelcar" or "native"

# Per-service override beats the cluster default.
apiVersion: serving.kserve.io/v1beta1
kind: InferenceService
metadata:
  name: llm-70b
spec:
  predictor:
    model:
      modelFormat:
        name: huggingface
      # oci://        -> follows ociModelMode (modelcar unless changed)
      # oci+native:// -> Kubernetes image volume, kubelet pulls
      # oci+fetch://  -> storage initializer pulls with oras-py
      storageUri: oci+native://registry.example.com/models/llm-70b-fp16:1.0

Note the status of oci+fetch://. The paper, written in July, calls it under review. The pull request was merged to KServe master on 12 August 2026. It streams layers into extraction, which the PR says cuts peak transient disk from about 2x the image to about 1x, but it rejected zstd layers in that PR, and it does not wire LLMInferenceService, which silently falls back to modelcar.

Integrity: hash while you stream, not after

OCI references can be pinned by digest and checked by an admission verifier, so the paper treats modelcar and oci+native:// as out of scope for its integrity work. The gap is s3://, gs://, hf:// and oci+fetch://. For those the author proposes digest pinning and OpenSSF model-signing enforcement at serving time, and measures what a SHA-256 check costs on a local object store.

The practical reading is that hashing inside the download loop is free and hashing afterwards re-reads every byte from a disk that is already the bottleneck. The paper does not draw conclusions from the 2 GB tier, which is page-cache dominated, and it has not yet measured the cost of keyless signature verification, the Fulcio and Rekor round trips.

The verification block below is the sketch in RFC #5789, which is still open, so none of these fields exist in a release. I include it because it shows the intended shape: a digest for single files, a signature bundle with an identity and issuer for model-signing, and enforce or warn modes set per namespace. The signing commands are the ones on the model-transparency project page.

# RFC kserve/kserve#5789 sketch. OPEN and not merged: these fields do not
# exist in any release yet.
model:
  storageUri: s3://bucket/model
  verification:
    digest: "sha256:..."            # single-file or archive artifacts only
    signature:
      bundle: "s3://bucket/model.sig"
      identity: "https://github.com/org/repo/.github/workflows/release.yml@refs/heads/main"
      oidcIssuer: "https://token.actions.githubusercontent.com"

# Signing and verifying with the OpenSSF model-signing CLI today:
model_signing sign bert-base-uncased --signature model.sig
model_signing verify bert-base-uncased --signature model.sig \
  --identity "$identity" --identity-provider "$oidc_provider"

Where it breaks: the failures the paper documents

The named failure in the paper is mundane. At 140 GB the storage initializer was OOM-killed until its memory limit was raised fourfold. Anyone moving big weights through the init container path should expect to tune that limit before anything else.

The image volume has its own failure mode in the Kubernetes docs: a failure to resolve or pull the image during pod startup blocks containers from starting, may add significant latency, and is retried with normal volume backoff. With Always the pod goes Failed if the pull fails, and with Never it goes Failed if any layer is not already present. A model volume with pullPolicy Never on an uncached node is a pod that cannot start.

One more trade deserves a line. A cold OCI pull at 140 GB took about 76 minutes in the paper's setup against about 39 for S3. If your scaling pattern is mostly brand-new nodes rather than new replicas on nodes that already hold the image, you are paying the 2x penalty every time and the warm benefit rarely shows up.

What the benchmark can and cannot tell you

The setup is one node, LAN-local, synthetic artifacts (a real scikit-learn model plus random ballast), with the registry and object store on the same machine. Multi-node distribution, registry rate limits and layer-parallel pulls are unmeasured. Time to first prediction measures artifact delivery only; GPU inference-engine initialisation is excluded, so a real LLM replica still pays CUDA graph capture and warmup on top of 11.7 s.

The harness is public under Apache 2.0, resumable, and scriptable down to the cluster. That matters more than the headline: the numbers are a hypothesis for your cluster, and the cheap way to test it is to rerun a smoke version with your own sizes and your own registry before you migrate anything.

What to do on Monday

A sequence that costs nothing before it commits you to anything.

1. Check the nodes: kubelet version at least 1.31 and a runtime that supports image volumes (containerd 2.1 or later, CRI-O 1.31 or later; subPath needs Kubernetes 1.33 and containerd 2.2).
2. Measure what you pay today: time a replica add from object storage for your largest model and note the free node disk it needs per concurrent replica.
3. Package one model as a single-layer image with the weights under models, push it, and reference it by digest rather than a mutable tag.
4. Enable enableOciModelSupport on a test cluster, deploy the same model with oci+native://, and time a cold start and then a second replica on the same node.
5. For s3 and hf sources you keep, hash while downloading rather than after, and plan to adopt digest pinning once RFC #5789 ships.

# 1. Kubelet and runtime versions per node.
kubectl get nodes -o custom-columns=NAME:.metadata.name,KUBELET:.status.nodeInfo.kubeletVersion,RUNTIME:.status.nodeInfo.containerRuntimeVersion

# 4. After deploying with oci+native://, confirm the cluster passed the
#    advisory check (OciImageVolumeCompatible) and read the pull events.
kubectl get inferenceservice llm-70b -o jsonpath='{.status.conditions}'
kubectl describe pod -l serving.kserve.io/inferenceservice=llm-70b | grep -iE 'pulling|pulled|image'

# Reproduce the paper on your own sizes: scripts are 01-cluster-up.sh to
# 04-integrity-bench.sh in the harness; override SIZES_GB and REPS to smoke test.
SIZES_GB="2" REPS=3 ./03-run-matrix.sh

If the second replica is not close to flat, the bottleneck is somewhere the paper did not measure, such as engine initialisation or a registry that cannot keep up, and the migration will not pay for itself.

Treat model weights as an image: pull once per node, address by digest, and mount instead of copying. The measured win is in the warm path, 11.7 s against 40.7 minutes, and it is paid for by a cold first pull that can double a plain download. Decide which of those two your autoscaler actually lives in before you choose the path.