Flux 2.9: CLI Plugins, SSA Ignore Rules and OIDC Webhooks

The Flux 2.9 announcement lists more than a dozen changes, and the headline one is easy to misread. A plugin system sounds like convenience. It is closer to a change in where GitOps tooling lives: validation, registry mirroring and operator debugging move out of ad hoc scripts and into the same binary you already run in CI.

I have not run 2.9 in production, so this is not a field report. It is a close reading of the announcement, the RFC behind the plugin system, the Kustomization and Receiver reference pages and the plugin READMEs, with every flag below taken from those pages. Where a source is silent, I say so instead of filling the gap.

The plugin system is a kubectl-style contract, with checksums

RFC-0013 models flux plugins on kubectl plugins: an executable named with the prefix flux- becomes a subcommand, so flux-operator is run as flux operator. The RFC says built-in commands win over plugins, nothing is scanned except one directory, and no plugin binary is executed during discovery, so a plugin cannot slow down flux commands that do not use it. Plugins are versioned independently of the CLI.

The install path is the part worth reading closely. The catalog lives in the fluxcd/plugins repository as YAML manifests with a per-platform download URL and a checksum. flux plugin install downloads to a temporary file, verifies the SHA-256 checksum, extracts only the declared binary, writes it with mode 0755 and records an install receipt beside it. The announcement recommends pinning by immutable digest for CI and production, and it names three plugins: Mirror, Schema and an Operator plugin maintained by ControlPlane.

# Discover, install, inspect. Binaries land in the plugin directory the
# RFC names (~/.fluxcd/plugins, override with FLUXCD_PLUGINS) as flux-<name>.
flux plugin search
flux plugin install schema          # latest: fine on a laptop
flux plugin list                    # versions and paths, read from install receipts

# Pin for CI and production. A version pin is the example in the announcement;
# a digest pin is immutable, so a re-tagged release cannot change what you run.
flux plugin install schema@0.5.0
flux plugin install schema@sha256:REPLACE_WITH_THE_DIGEST_FROM_THE_CATALOG

flux plugin update schema
flux plugin uninstall schema

One discrepancy to know about: the announcement says binaries land in a fluxcd/plugins folder under the home directory, while the RFC specifies ~/.fluxcd/plugins with a FLUXCD_PLUGINS override. I would trust the RFC and confirm with flux plugin list, which prints each plugin's path.

Flux Schema turns manifest validation into a CI step

The Schema plugin validates Kubernetes YAML against JSON Schema and CEL rules, and its README says it reuses the API server's evaluation semantics for x-kubernetes-validations. Its built-in default catalog covers Kubernetes, OpenShift, Gateway API, Flux, Flagger and Flux Operator CRDs, and an ecosystem catalog adds schemas for other CNCF projects. That means a pull request can fail on a bad HelmRelease or Kustomization before a controller ever sees it.

The README ships a minimal workflow, reproduced below, and a Docker image that carries the default catalog at /catalog/latest so validation also works in an air-gapped runner. I pinned nothing in the workflow because I copied the README; in production I would replace main with a tag or commit, for the same reason the announcement pins plugins by digest.

# .github/workflows/flux-schema.yaml (from the flux-schema README)
name: flux-schema

on:
  pull_request:
    branches: [main]

jobs:
  validate:
    runs-on: ubuntu-latest
    steps:
      - name: Checkout
        uses: actions/checkout@v7
      - name: Setup Flux CLI with Schema Plugin
        uses: fluxcd/flux2/action@main     # production: pin a tag or commit
        with:
          plugins: |
            schema
      - name: Validate manifests
        uses: fluxcd/flux-schema/actions/validate@main

# Air-gapped runner: the image carries the default catalog.
docker run --rm \
  -v "$PWD/manifests:/manifests:ro" \
  ghcr.io/fluxcd/flux-schema:latest validate /manifests \
  --schema-location /catalog/latest

# Locally, including a Helm chart piped through the validator.
flux schema validate ./clusters/production -s ecosystem -v
helm template ./charts/app | flux schema validate -v --skip-missing-schemas

The README is brief on writing your own CEL rules and points elsewhere for that, so I am not claiming how custom rules are authored. What it does document is flux schema discover, which catalogs the Flux and Kubernetes resources in a GitOps repository, and flux schema extract crd, which builds a catalog from your own CRDs.

Flux Mirror takes the upstream registry out of the reconcile path

Mirror copies Helm charts, OCI artifacts and container images from one registry to another from a declarative config. The README frames the goal as feeding an internal registry so clusters never pull from upstream at reconcile time, and as a way to move charts off HTTP/S HelmRepository sources onto OCI. Re-runs only copy missing or drifted tags, and drift is reported through its own exit code.

# flux-mirror.yaml (shape from the flux-mirror README)
apiVersion: mirror.plugin.fluxcd.io/v1beta1
kind: Config
charts:
  - name: external-dns
    source: https://kubernetes-sigs.github.io/external-dns/
    destination: oci://ghcr.io/my-org/charts
    version: "*"
    limit: 3                      # keep the newest three, not the whole history
artifacts:
  - source: registry.k8s.io/external-dns/external-dns
    destination: ghcr.io/my-org/external-dns
    selector:
      semver: ">=0.15.0"
      limit: 3
    includeReferrers: true        # signatures, SBOMs and attestations travel too

# Run it on a schedule and alert on the exit code. A mirror that stopped
# syncing looks exactly like a mirror that is up to date.
#   flux mirror sync flux-mirror.yaml

Filtering for artifacts is a four-step pipeline of regex, semver, sort and top-N. Authentication covers Docker config and credential helpers, cloud workload identity for ECR, ACR and GAR, and several bearer sources. Signature verification is opt-in and checks keyless signatures on the source artifact before copying, with an optional minimum signature age.

The trade-off is that a mirror is infrastructure you now operate. If the sync job stops, your internal registry silently falls behind upstream, and a security fix upstream does not reach the cluster. Run flux mirror sync on a schedule and alert on its exit code rather than treating the first successful run as done.

SSA ignore rules let another controller own a field

Before 2.9, an autoscaler and Flux fighting over spec.replicas was a classic GitOps annoyance. Kustomization now has spec.ignore, a list of rules with required JSON Pointer paths and an optional target selector. The reference page says ignored fields are excluded from drift detection and correction, and it describes two strategies: strip, which removes the field from the apply payload when another server-side apply manager owns it, and adopt, which copies the in-cluster value into the payload when Flux is the only apply manager.

The target accepts regular expressions for group, version, kind, name and namespace, plus label and annotation selectors. In paths, a slash inside a key must be written as ~1, which matters for annotation names. The example below is mine, built from those documented fields; it ignores replicas on every Deployment and one external-dns annotation on a single Service.

apiVersion: kustomize.toolkit.fluxcd.io/v1
kind: Kustomization
metadata:
  name: apps
  namespace: flux-system
spec:
  interval: 10m
  path: ./apps/production
  prune: true
  sourceRef:
    kind: GitRepository
    name: flux-system
  ignore:
    # The HPA owns replicas. Scoped to Deployments only.
    - paths:
        - /spec/replicas
      target:
        kind: Deployment
    # external-dns writes this annotation. "/" inside a key is escaped as "~1".
    - paths:
        - /metadata/annotations/external-dns.alpha.kubernetes.io~1hostname
      target:
        kind: Service
        name: web

# Wrong: no target. This rule now matches EVERY object in the Kustomization.
#   ignore:
#     - paths:
#         - /spec/replicas

Two sentences from the reference page deserve to be read twice. Omitting target makes the rule match every object the Kustomization manages, and changes to ignored fields alone do not trigger a reconciliation. Both are correct behaviour, and both will surprise someone who expected a manual edit to be corrected at the next interval.

OIDC Receivers remove the shared webhook secret

A generic Receiver has always been protected by a token. The new generic-oidc type verifies an OIDC ID token sent as an Authorization Bearer header instead, so CI holds no stored secret. The controller matches the token's iss claim to a configured provider, checks signature, expiry and audience (default notification-controller), then evaluates your CEL validations. Every one must be true or the request is rejected and the failing message is returned.

The same release adds per-resource filtering: each entry in spec.resources can carry a CEL filter, and it stacks with the top-level resourceFilter, so a resource is reconciled only when both return true. There is also a new flux trigger receiver command that sends the request for you, with --type generic-oidc and --oidc-provider github or forgejo. On GitHub the job needs the id-token write permission.

apiVersion: notification.toolkit.fluxcd.io/v1
kind: Receiver
metadata:
  name: platform-deploy
  namespace: flux-system
spec:
  type: generic-oidc               # spec.secretRef is rejected for this type
  oidcProviders:
    - issuerURL: https://token.actions.githubusercontent.com   # must equal the token's iss claim
      # audience defaults to notification-controller
      validations:
        # A valid signature alone authorises nothing: pin the identity.
        - expression: "claims.repository_owner == 'my-org' && claims.repository == 'my-org/platform'"
          message: "only my-org/platform may trigger this Receiver"
  resources:
    - apiVersion: source.toolkit.fluxcd.io/v1
      kind: GitRepository
      name: flux-system

# In the GitHub Actions job (needs: permissions: id-token: write)
#   flux trigger receiver platform-deploy \
#     --type generic-oidc --oidc-provider github \
#     --url https://flux-hooks.example.com -n flux-system

The reference page is blunt about the risk: a valid signature does not authorise anything by itself. Public issuers such as GitHub Actions hand tokens to any workflow on the platform, so your validations must pin identity claims such as repository_owner and repository. Event filtering through spec.events is not supported for this type, and an invalid CEL expression marks the Receiver Stalled with reason InvalidCELExpression and rejects every request.

Post-quantum age keys and secret-less Vault in the decrypt path

The release notes list SOPS decryption with the Age post-quantum cipher for Kustomization, and give no more detail: no SOPS or age version numbers. For the key side, the age README says hybrid post-quantum keys are generated with age-keygen -pq, are built into age v1.3.0 and later, start with AGE-SECRET-KEY-PQ-1 for the identity and age1pq1 for the recipient, and that recipients are about 2000 characters long.

On the Flux side nothing changes in shape: the Kustomization decryption block still names provider sops and a Secret, and an age identity goes in a key whose name ends in .agekey. What I could not confirm from the pages I read is which SOPS release you need to encrypt with a post-quantum recipient, because the SOPS README is silent on it. Test encrypt and decrypt in a scratch repository before you commit the first real secret.

# 1. A hybrid post-quantum age key (age v1.3.0 or later, per the age README).
age-keygen -pq -o key.txt
age-keygen -y key.txt > recipient.txt      # age1pq1..., about 2000 characters

# 2. Give the controller the identity. The key name must end in .agekey.
kubectl -n flux-system create secret generic sops-age \
  --from-file=identity.agekey=key.txt

# 3. Point the Kustomization at it.
#   spec:
#     decryption:
#       provider: sops
#       secretRef:
#         name: sops-age
#
# Leave metadata, kind and apiVersion of the encrypted Secret in plain text.
# Test encrypt and decrypt in a scratch repository first: the SOPS README is
# silent on which release accepts a post-quantum recipient.

The second secrets change is Workload Identity for OpenBao and HashiCorp Vault. The reference page explains that kustomize-controller exchanges its ServiceAccount token for a short-lived Vault token through a JWT-backed auth method, enabled with the --sops-vault-configmap flag. That ConfigMap doubles as an allowlist of Vault addresses, the token audience is the Vault address, and the Vault role is derived as namespace_name from the ServiceAccount. It also requires the ObjectLevelWorkloadIdentity feature gate when you use serviceAccountName.

What breaks when you upgrade, and what does not

The announcement has a short list of changes that can fail a reconcile after upgrade. They are all knowable in advance, which is why I would put them in a pre-flight checklist rather than discover them from a stalled Kustomization.

The plugin system itself carries limits the RFC lists for v1beta1: checksums are SHA-256 only with Cosign and SLSA deferred, no plugin dependencies or version ranges, no automatic update checks and no authentication for private catalogs. Flag names can also differ between Flux and a plugin, the RFC's own example being --context against --kube-context. None of this is a reason to avoid plugins, but it is a reason to pin digests and treat plugin updates as a reviewed change.

The 2.8 release belongs in the same conversation. It made Helm v4 behaviour, with server-side apply and kstatus health checks, the default for new HelmReleases while existing ones keep client-side apply until you opt in, and its UseHelm3Defaults feature gate restores the previous defaults. If you are jumping from before 2.8, read both announcements and the upgrade procedure discussion the 2.9 post links to.

What to do on Monday

Do these in order. Steps one and two protect the upgrade, step three changes nothing in the cluster, and steps four and five are the real adoption work.

1. Find resources on removed API versions and run flux migrate against them before touching the controllers.
2. Set the Helm post-render strategy explicitly on any HelmRelease whose chart ships hooks, then upgrade and compare rendered output.
3. Add flux schema validate to a pull request workflow in report-only mode, and read what it flags in your repository for a week.
4. Replace one HPA-versus-Git conflict with a targeted spec.ignore rule, with kind set, and confirm the field is no longer reverted.
5. Move one webhook from a token to generic-oidc with pinned repository claims, then delete the stored secret from CI.

# 1. Resources still on the APIs removed in 2.9.
grep -rn -E "(image|notification)\.toolkit\.fluxcd\.io/v1beta2" clusters/ apps/
flux migrate --help                  # then migrate to the stable APIs

# 2. Every HelmRelease, to find the ones whose charts ship hooks.
kubectl get helmreleases -A

# 3. Server version against the supported minimums: 1.34.1, 1.35.0, 1.36.0.
kubectl version

# 4. Read-only adoption: validate without blocking merges (continue-on-error
#    on the workflow step) and look at what it reports.
flux plugin install schema@sha256:REPLACE_WITH_THE_DIGEST_FROM_THE_CATALOG
flux schema validate ./clusters/production -s ecosystem -v

Steps one and two are the ones that prevent an incident. Steps three to five are improvements you can take at your own pace, and each can be reverted by deleting a file or a field.

Flux 2.9 is best read as a set of small boundaries redrawn: the CLI can grow without the core growing, Flux can leave a field to someone else, and a webhook can trust an identity instead of a string. Pin what you install, scope every rule with a target or a claim, and read the post-render default before you upgrade.