AI & DevOps · Kubernetes
Kubernetes for AI, in reading order
The articles on this site about running AI workloads on Kubernetes, arranged as one path: upgrade the cluster, give it GPUs it can schedule, route model traffic through Gateway API, then let agents work inside it safely.
By Matthews WongReviewed on
Key takeaway
This path starts with what the 1.36 and 1.37 releases change for platform teams, then covers GPU allocation with DRA, gang scheduling and the KAI and Volcano schedulers, then the move from Ingress to Gateway API and its inference extension, and ends with agents on the cluster: the Agent Sandbox resource, kagent, and read-only MCP access.
Reading path · 24 articles
Read in this order
Each stage builds on the one before. If you already know a stage, skip ahead; the line under each title says what that article adds.
Stages 1 / 4
Upgrade first: what 1.36 and 1.37 change
The release changes that land on every cluster before any AI feature does.
- 01DevOps12 min readKubernetes 1.37 Release: What Breaks When Upgrading From 1.36Why this step: The 1.37 overview for platform teams: defaults that flip on, deprecation dates and a preflight script.
- 02Security12 min readKubernetes 1.36 Security Changes: userns, kubelet, gitRepoWhy this step: The 1.36 security changes: user namespaces GA, fine-grained kubelet authorization and gitRepo removed, with an audit script.
- 03Security12 min readKubernetes Manifest-Based Admission Control Beta in 1.37Why this step: Admission webhooks and CEL policies loaded from disk, so they still apply when etcd is down.
- 04Security10 min readcontainerd runAsNonRoot Bypass: CVE-2026-46680 Fix and AuditWhy this step: A crafted image could defeat runAsNonRoot; the fixed containerd versions, the Pod mitigation and a cluster audit.
- 05DevOps11 min readKYAML in Kubernetes 1.37: Stable YAML Subset, kubectl -o kyamlWhy this step: KYAML is stable: what the strict YAML subset enforces and how to use it in CI.
- 06DevOps12 min readkube-proxy ipvs Deprecated: Migrate to nftables by v1.40Why this step: kube-proxy's ipvs mode is deprecated; a sourced nftables migration plan with kernel checks.
Stages 2 / 4
GPUs: DRA, gang scheduling and AI conformance
How a cluster hands out accelerators, starts multi-pod jobs together, and proves it is ready for AI work.
- 07DevOps12 min readKubernetes DRA GPU Fallback: Prioritized List in 1.36Why this step: Ask for one GPU type and fall back to another with the DRA prioritized list, plus MIG partitions and device taints.
- 08DevOps12 min readKubernetes 1.37 DRA Extended Resources GA: Run example.com/gpu Without a Device PluginWhy this step: Leave the device plugin without rewriting manifests: map one DeviceClass and migrate nodes one by one.
- 09DevOps12 min readKubernetes Gang Scheduling Beta in 1.37: What Works, What NotWhy this step: Native gang scheduling for training jobs: the Workload and PodGroup APIs, and what still needs Kueue or Volcano.
- 10DevOps12 min readVolcano Scheduler GPU Gang Preemption: v1.15 DRA Queue QuotaWhy this step: Volcano's gang preemption, DRA queue quota and scheduling gates, set against the native gang scheduling.
- 11DevOps11 min readNVIDIA Run:ai Open Source: KAI Scheduler, GPU FractioningWhy this step: The open-sourced Run:ai parts: KAI Scheduler, GPU fractioning, the CUDA catch and the licence terms.
- 12DevOps12 min readKubernetes HPA Scale to Zero for GPU Queue Workers (1.37)Why this step: Scale GPU queue workers to zero with the native HPA, and the trap to avoid.
- 13AI12 min readKubernetes AI Conformance: What the KARs Demand of a PlatformWhy this step: Check a platform against the AI conformance requirements, and what the badge does not prove.
Stages 3 / 4
Traffic: Gateway API and inference routing
Off Ingress and onto Gateway API, then the extension that routes requests to model servers.
- 14DevOps9 min readKubernetes Gateway API vs Ingress: Why It Replaces AnnotationsWhy this step: The groundwork: how Gateway API's role-oriented resources replace Ingress.
- 15DevOps12 min readingress-nginx Retirement: Contour or Gateway API MigrationWhy this step: With ingress-nginx retired, compare a Contour lift-and-shift with a move to Gateway API.
- 16DevOps12 min readIngress2Gateway 1.0: Migrate ingress-nginx to Gateway APIWhy this step: The hands-on conversion with Ingress2Gateway: read its warnings and fix what it cannot translate.
- 17DevOps11 min readGateway API Inference Extension: EPP Moves to llm-d, What NowWhy this step: Gateway API for model traffic: what the Inference Extension keeps upstream now the Endpoint Picker lives in llm-d.
- 18DevOps12 min readllm-d on Kubernetes: Distributed Inference and v0.10 HardeningWhy this step: The serving stack built on that extension, and what its hardening release changes for rollouts.
Stages 4 / 4
Agents on the cluster
Running agents as Kubernetes workloads, and giving them cluster access that cannot break it.
- 19AI12 min readKubernetes Agent Sandbox CRD: Isolated Stateful AI AgentsWhy this step: Isolated, stateful agents as a Kubernetes resource: Sandbox, templates, claims and warm pools.
- 20AI12 min readkagent Kubernetes Tutorial: AI Agents on a Laptop ClusterWhy this step: Run AI agents on a laptop kind cluster with kagent and wire MCP tools into them.
- 21DevOps12 min readKubernetes MCP Server: read_only, Toolsets, Denied ResourcesWhy this step: Give an agent cluster access that cannot uninstall anything: a view ServiceAccount, read-only mode and denied resources.
- 22DevOps12 min readkubectl-ai Guide: Confirmation Gate, Limits and MCP Server RiskWhy this step: How kubectl-ai's confirmation gate classifies commands, and why its MCP HTTP server needs OAuth.
- 23DevOps11 min readAI DevOps Assistants for Kubernetes Production TriageWhy this step: What an AI assistant really shortens in Kubernetes triage, and the calls it should not make for you.
- 24DevOps12 min readKubeCon North America 2026 Guide: AI Track, Co-located DaysWhy this step: To go further in person: a platform engineer's guide to KubeCon North America and its AI Inference and Agentic track.
The AI & DevOps field guide
More from the AI & DevOps guide
The other reading paths in this section.
- MCP & Agent SecurityPrompt injection, the 2026 MCP incidents, access controls, sandboxes and vetting agent skills.
- LLM Inference ServingvLLM and SGLang, disaggregated serving, llm-d and KServe, autoscaling and cost per token.
- AI SRE & Agentic OpsIncident-investigation agents, MCP servers for ops tools, agents in CI, and the IaC beneath.
- Supply Chain & Regulationnpm and PyPI worms, model signing and AIBOM, secrets in agent commits, the CRA and EU AI Act.