Kubernetes Manifest-Based Admission Control Beta in 1.37

Every admission policy I have ever deployed to a cluster shared one weakness: it was an API object, so it only existed after the API server was up and etcd had answered, and it could be deleted by anyone whose RBAC allowed it. During a restore from backup, that is the window where the policy you rely on is simply not there.

Kubernetes 1.37 (Garhwal, released 26 August 2026) graduates manifest-based admission control to Beta and enables it by default. This post is a close reading of the official docs page, the v1.36 alpha announcement, the 1.37 release post and KEP-5793. I have not run it on a production control plane, so everything below is what those documents state, plus config I assembled from their examples and say so where I did.

Three gaps that API-based admission cannot close

The docs name three limits of policies registered through the API. The bootstrap gap: the API objects must be created and loaded by the dynamic admission controller, and until that happens nothing is enforced. The self-protection gap: admission configuration resources are not themselves subject to webhook admission, to avoid circular dependencies, so a sufficiently privileged user can delete or modify a critical policy. The etcd dependency: if etcd is unavailable or corrupted, API-based policies may not load correctly.

The v1.36 announcement puts the first gap in operational terms: when you restore from a backup or recover from an etcd failure, the window between the API server serving requests and your policies becoming active can be significant. KEP-5793 lists exactly this as a user story: a restored cluster, an etcd that has been reset, or a brand-new bootstrap, where a security webhook must be called from the first request.

The KEP also explains why the usual workarounds fall short. External configuration management such as Helm, Kustomize or GitOps does not eliminate the bootstrap gap and the configurations can still be deleted through the API. RBAC deny rules would need a significant redesign. Compiling a custom admission plugin into the API server needs custom builds. A file read by the API server itself sidesteps all three.

How staticManifestsDir wires it up

The mechanism is one field. You add staticManifestsDir to the plugin configuration of an AdmissionConfiguration file, and pass that file to kube-apiserver with --admission-control-config-file. The ManifestBasedAdmissionControlConfig feature gate must be on; per the docs it has been enabled by default since 1.37, and it was alpha and off by default in 1.36. Each of the four plugins, ValidatingAdmissionWebhook, MutatingAdmissionWebhook, ValidatingAdmissionPolicy and MutatingAdmissionPolicy, needs its own directory.

The config below is the docs example, trimmed to the two plugins I would start with. The kube-apiserver fragment underneath is my own illustration of a kubeadm-style static pod, not something the docs prescribe; the v1.36 post only says that if the API server runs as a Pod you need to mount the directory in, and that read-only is fine.

# /etc/kubernetes/admission/admission-config.yaml
# Passed to kube-apiserver with --admission-control-config-file.
# Absolute paths only: no globs, no relative paths.
apiVersion: apiserver.config.k8s.io/v1
kind: AdmissionConfiguration
plugins:
- name: ValidatingAdmissionPolicy
  configuration:
    apiVersion: apiserver.config.k8s.io/v1
    kind: ValidatingAdmissionPolicyConfiguration
    staticManifestsDir: "/etc/kubernetes/admission/validating-policies/"
- name: ValidatingAdmissionWebhook
  configuration:
    apiVersion: apiserver.config.k8s.io/v1
    kind: WebhookAdmissionConfiguration
    kubeConfigFile: "<path-to-kubeconfig>"
    staticManifestsDir: "/etc/kubernetes/admission/validating-webhooks/"

# My illustration, not from the docs: a kubeadm-style static pod fragment.
# The directory must be visible INSIDE the kube-apiserver container;
# read-only is enough because the API server only reads it.
#   command:
#   - kube-apiserver
#   - --admission-control-config-file=/etc/kubernetes/admission/admission-config.yaml
#   volumeMounts:
#   - name: admission
#     mountPath: /etc/kubernetes/admission
#     readOnly: true
#   volumes:
#   - name: admission
#     hostPath:
#       path: /etc/kubernetes/admission
#       type: Directory

Three details decide whether this works on the first try. The path must be absolute; glob patterns and relative paths are not supported. Only direct-child files ending in .yaml, .yml or .json are loaded, and subdirectories and other extensions are ignored. And each directory may only hold resource types its plugin accepts, so a ValidatingAdmissionPolicy directory can contain policies and bindings and nothing else.

What changes versus an API-based policy

The trade is isolation for flexibility. A manifest-based configuration lives in what KEP-5793 calls an isolated universe: it may not reference API resources, nor the reverse. That one rule explains every restriction in the table below, because a policy that needs a ConfigMap or a Service has a dependency on the very cluster state that might not exist yet at startup.

Manifest files go through the strict decoder, which rejects duplicate and unknown fields, and each object gets the same defaulting and validation as the REST API. You can group a policy and its binding with a v1.List or with --- separators, and the docs note a policy and its binding are loaded together atomically.

Protecting the protectors

The headline capability is the one API-based admission cannot offer: a policy that intercepts operations on ValidatingAdmissionPolicy, MutatingAdmissionPolicy, their bindings and the two webhook configuration types. The docs example denies DELETE and UPDATE on any of them that carries the platform.example.com/protected label set to true. The expression passes unless the old object has that label, which is why it checks oldObject rather than object.

Save it as a file in the ValidatingAdmissionPolicy directory. The names end in .static.k8s.io, failurePolicy is Fail, and the binding uses the Deny validation action. This is the docs example, reproduced with its names.

# /etc/kubernetes/admission/validating-policies/protect-admission-resources.yaml
apiVersion: admissionregistration.k8s.io/v1
kind: ValidatingAdmissionPolicy
metadata:
  name: "example-protect-admission-resources.static.k8s.io"   # suffix is mandatory
spec:
  failurePolicy: Fail
  matchConstraints:
    resourceRules:
    - apiGroups: ["admissionregistration.k8s.io"]
      apiVersions: ["*"]
      operations: ["DELETE", "UPDATE"]
      resources:
      - "validatingadmissionpolicies"
      - "validatingadmissionpolicybindings"
      - "mutatingadmissionpolicies"
      - "mutatingadmissionpolicybindings"
      - "validatingwebhookconfigurations"
      - "mutatingwebhookconfigurations"
  validations:
  # oldObject, not object: on DELETE there is no new object to inspect.
  - expression: >-
      !has(oldObject.metadata.labels) ||
      !('platform.example.com/protected' in oldObject.metadata.labels) ||
      oldObject.metadata.labels['platform.example.com/protected'] != 'true'
    message: "Protected admission resources cannot be modified or deleted"
---
apiVersion: admissionregistration.k8s.io/v1
kind: ValidatingAdmissionPolicyBinding
metadata:
  name: "example-protect-admission-resources-binding.static.k8s.io"
spec:
  # Must name a policy in the same manifest set, with the same suffix.
  # spec.paramRef is not allowed here.
  policyName: "example-protect-admission-resources.static.k8s.io"
  validationActions:
  - Deny

Order matters here too. Manifest-based configurations are evaluated before API-based ones, so platform policy wins over anything a team registers later. For the admission resource types themselves, only manifest-based hooks run; API-based hooks are skipped. TokenReview and SubjectAccessReview are excluded for everyone, because intercepting them could lock the cluster out of its own authentication and authorisation path.

Reload semantics: strict at boot, forgiving at runtime

Startup and runtime follow opposite rules, deliberately. At startup every manifest is read and validated, the API server does not become ready until all load successfully, and any invalid manifest makes it fail to start. The v1.36 post defends this: failing fast is safer than running without the policies you expected. The same applies if two files define the same type and name.

At runtime the directories are watched with fsnotify and a polling fallback, default one minute. A content hash of all manifest files is computed on each check and nothing happens if it is unchanged. A new configuration is validated first; if it fails, the error is logged, metrics are updated and the previous valid configuration stays. A successful reload replaces the old one atomically.

The docs ask you to write changes atomically, for example to a temporary file and then rename, so the watcher never reads a half-written file. This matters most for mounted ConfigMaps and Secrets. Because files without a .yaml, .yml or .json extension are ignored, a staging name with a different extension is invisible to the loader until the rename.

# Wrong: edit in place. The watcher can read a half-written file,
# fail validation, and keep the OLD policy while you think the new one is live.
vi /etc/kubernetes/admission/validating-policies/deny-privileged.yaml

# Right: stage under a name the loader ignores (only .yaml, .yml and .json
# are loaded), then rename. A rename within one filesystem is atomic.
D=/etc/kubernetes/admission/validating-policies
cp deny-privileged.yaml "$D/deny-privileged.yaml.staging"
mv "$D/deny-privileged.yaml.staging" "$D/deny-privileged.yaml"

# Then confirm the reload actually succeeded. Do not trust the mv.
# Metric names are from the docs; the 1 minute poll is the default fallback.
curl -sk --cert "$CERT" --key "$KEY" https://127.0.0.1:6443/metrics \
  | grep apiserver_manifest_admission_config_controller

Where it bites: the trade-offs the docs spell out

KEP-5793 lists its own drawbacks without softening them: reduced visibility, because you cannot list active admission controls through the API; operational complexity in HA clusters; harder debugging, because a denial may come from a file you cannot kubectl get; and limited functionality. The docs troubleshooting table turns these into symptoms. Two more I would put on a rollout checklist come from elsewhere on the page: downgrading with staticManifestsDir still configured makes the API server fail to start on unknown fields, and in 1.37 a webhook that intercepted authentication or authorisation review requests silently stops receiving them.

The last row deserves its own sentence because it is not about manifests at all. KEP-5793 bundles a behaviour change into the beta: ValidatingAdmissionWebhook and MutatingAdmissionWebhook now skip the same authentication and authorisation virtual resources that the CEL policies already skipped. It is a short-term opt-out, will be locked at GA, and the KEP says a deprecation warning fires on writes of webhook rules that name those resources. Audit your webhook rules before upgrading, not after.

Observability: you cannot kubectl get it

Since nothing is visible through the API, the docs lean on three signals. Metrics: a reload counter with status, plugin and apiserver_id_hash labels, a last-reload timestamp gauge, and a config info gauge whose hash label exposes drift between API servers. Existing admission metrics such as apiserver_admission_webhook_rejection_count carry the object name, so the .static.k8s.io suffix identifies manifest-based decisions. Audit annotations include the object name too. The rules below are my own PromQL built on those metric names, to adapt rather than copy.

# Alert 1: a reload was rejected, so the previous policy is still running.
- alert: ManifestAdmissionReloadFailed
  expr: increase(apiserver_manifest_admission_config_controller_automatic_reloads_total{status="failure"}[10m]) > 0
  labels:
    severity: page

# Alert 2: control plane nodes disagree about which files are loaded.
# last_config_info is always 1 and carries the hash as a label.
- alert: ManifestAdmissionConfigDrift
  expr: count by (plugin) (count by (plugin, hash) (apiserver_manifest_admission_config_controller_last_config_info)) > 1
  for: 15m
  labels:
    severity: ticket

# Who said no? Manifest-based decisions carry the .static.k8s.io suffix.
# sum by (name) (rate(apiserver_admission_webhook_rejection_count{name=~".*\.static\.k8s\.io"}[5m]))

The second rule works because the config info value is always 1 and carries the hash as a label: if the count of distinct hashes for a plugin is above one, at least two API servers hold different files. Run it with a for clause long enough to ride out a rolling file push, since servers reload on their own schedule.

What to do on Monday

Start with one policy you would be embarrassed to lose, and treat it as a change to the control plane, not to the cluster:

1. Check that you control kube-apiserver flags and its filesystem. I did not verify whether any managed Kubernetes provider exposes --admission-control-config-file, so confirm that before planning.
2. Rehearse on a throwaway 1.37 cluster: a broken file must stop the API server from starting, and a broken edit must leave the old policy running with the failure counter incremented.
3. Start with the deny-privileged policy from the docs, then add the protect-admission-resources policy once you have decided which labels mark protected objects.
4. Distribute the files to every control plane node with configuration management, and alert on the reload failure counter and on more than one config hash per plugin.
5. Write the downgrade note now: remove staticManifestsDir from the AdmissionConfiguration and restart before moving a control plane to a version without the feature.

The checks below prove both halves of the feature on a test cluster. The first applies a privileged pod outside kube-system, which the deny-privileged policy should refuse with its own message. The second tries to delete an admission policy labelled protected, which the protect policy should refuse. Neither command needs the policy to exist in etcd, which is the point.

# 1. Privileged pod outside kube-system: the deny-privileged policy should refuse it.
kubectl create namespace policy-test
cat <<'EOF' | kubectl apply -n policy-test -f -
apiVersion: v1
kind: Pod
metadata: { name: priv }
spec:
  containers:
  - name: c
    image: busybox:1.36
    command: ["sleep", "3600"]
    securityContext: { privileged: true }
EOF
# expect: Privileged containers are not allowed

# 2. Delete a policy labelled protected: the protect policy should refuse it.
kubectl label validatingadmissionpolicy my-baseline platform.example.com/protected=true
kubectl delete validatingadmissionpolicy my-baseline
# expect: Protected admission resources cannot be modified or deleted
# (the label UPDATE itself passes: oldObject had no label yet)

The rule I take from this: a policy that must survive a mistake belongs outside the system it governs. Manifest-based admission control makes that possible for admission itself, at the cost of treating policy as control-plane configuration that you distribute, watch and downgrade by hand. Use it for the handful of rules whose absence would hurt, and leave everything parameterised or team-owned in the API.