HolmesGPT Kubernetes Alerts: Runbooks Beat the Model Choice

The number that stopped me was not the cost. It was 16 to 2. Before the STCLab team wrote exclusion rules, each HolmesGPT investigation wasted 16 tool calls on systems the cluster does not run. After, it wasted two. No model swap in their write-up moved a number that far.

This post reads the CNCF write-up from April 2026 by Grace Park and Ihyeok Song closely, then checks each mechanism against the HolmesGPT docs I fetched on 2026-10-17. I have not run their pipeline. Where the article is silent, such as the contents of the seven runbooks, I say so and mark my own sketches as mine.

Alert to Slack thread: the pipeline in five stages

STCLab's SRE team has two engineers and runs several Amazon EKS clusters with high-traffic production workloads. Their observability stack is OpenTelemetry feeding Mimir, Loki and Tempo, and Robusta OSS already enriched Prometheus alerts and posted them to Slack. Manual triage took 15 to 20 minutes per alert, so the goal was a first pass written into the same Slack thread.

Two numbers describe the output. About 40 raw alerts a day collapse into about 12 unique investigations, and an engineer reads a threaded summary in under two minutes instead of spending 15 to 20 minutes on manual triage. About 40 percent of investigations resolve themselves, mostly known patterns such as OOMKilled and ImagePullBackOff that a runbook matches. All of these figures are the team's own, reported once, with no independent measurement.

Runbooks moved the score more than the model did

The article's strongest evidence is a controlled comparison on the same model. On a ClickHouse handshake alert, Holmes with runbooks matched the known pattern in 3 to 4 tool calls. Without them it chased three unrelated hypotheses and used 20 or more steps. A 12-step budget was enough with runbooks.

Read the last row carefully: it compares before and after exclusion rules, not runbooks against none, and the article attributes the gain to rules like no Loki, no Tempo and no Istio here. The article does not describe how the 4.6 and 3.6 scores were produced, so treat them as a direction, not a benchmark. HolmesGPT's own docs describe 150 plus evals scored against a list of expected statements, which is the kind of harness you would need to reproduce this on your own alerts.

What a runbook actually contains

The article says each runbook is a markdown file with a Meta header holding scope, tools and caution fields, and that Holmes calls fetch_runbook early to learn which tools to use or skip. Its examples of scope and caution are a namespace-only scope and a note that some containers are excluded from log collection. It does not publish the seven runbooks, so the file below is my reconstruction from those described fields, not theirs.

## Meta
scope: namespace=payments only
tools: kubernetes, prometheus
caution: the batch-export containers are excluded from log collection,
         so an empty Loki result for them means nothing

## Do not use
Loki, Tempo and Istio are not deployed in this cluster. Do not query them.

## Alert: KubePodCrashLooping
1. Read the last termination reason from the pod status.
2. OOMKilled: compare the memory limit with the working set in Prometheus.
3. ImagePullBackOff: read the image tag and the pull secret, nothing else.
4. Stop at three tool calls unless the reason is none of the above.

In current Holmes the same knowledge is a skill: a directory with a SKILL.md, YAML frontmatter with a required description, and recommended sections named Goal, Workflow, Synthesize Findings and Recommended Remediation Steps. The description is what the model matches against the alert, so it must name a concrete problem. Here is the same crash-loop runbook in that shape, again my own example.

---
name: payments-crashloop-triage
description: Pods crash-looping in the payments namespace on the production cluster
last_updated: 2026-10-17
---

## Goal
Name the termination reason and the one change that fixes it.

## Workflow
1. Do not query Loki, Tempo or Istio: none are deployed here.
2. Read the pod status for the last termination reason.
3. OOMKilled: compare the memory limit with usage in Prometheus.

## Synthesize Findings
State the reason, the evidence, and what you did not check.

## Recommended Remediation Steps
- Owner: payments team, #payments-oncall. Escalate to the SRE lead.

Notice what both files spend their words on: what to skip, when to stop and who owns the problem. None of it is clever prompting. It is the knowledge a senior engineer carries about this one cluster, written where the agent reads it before it picks a tool.

Runbooks became skills: what changes if you start today

The current HolmesGPT docs call skills the successor to runbooks, available from Holmes 0.26.0, with earlier versions using the legacy runbook system. I could not confirm which version STCLab ran, and the docs page does not mention fetch_runbook at all, so I treat the two as the same idea in two generations rather than claiming how they map one to one.

The practical upshot is that skills live in a Git repo with a review process, which is where a seven-file knowledge base belongs. The docs add that Holmes can draft a skill from a finished investigation, but the output needs review before it is committed, because unreviewed generated skills may contain incorrect steps.

The glue code is where the pipeline earns its keep

STCLab's lesson here is that Holmes handles reasoning and about 200 lines of Python handle everything else: finding the Slack thread Robusta already created, replicating its namespace-to-channel routing, and fingerprinting alerts at the workload level so repeats are suppressed for 30 minutes during rollouts. The workload-level key matters because a rollout renames every pod. This is my sketch of that rule, not their playbook.

# My sketch of the dedup rule, not STCLab's code. Their playbook is about
# 200 lines of Python inside Robusta and also finds the Slack thread and
# replicates the namespace-to-channel routing.
import time

SUPPRESS_SECONDS = 30 * 60  # the 30-minute window from the CNCF write-up
_last_seen: dict[str, float] = {}


def workload_fingerprint(labels: dict[str, str]) -> str:
    # Wrong: key on the pod name. A rollout renames every pod, so every
    # replacement looks like a brand-new alert and the window never holds.
    # Right: key on the owning workload plus the alert name.
    workload = (
        labels.get("deployment")
        or labels.get("statefulset")
        or labels.get("daemonset")
        or labels.get("pod", "unknown")
    )
    return "|".join([labels.get("namespace", "-"), workload, labels["alertname"]])


def should_investigate(labels: dict[str, str], now: float | None = None) -> bool:
    now = time.time() if now is None else now
    fp = workload_fingerprint(labels)
    if now - _last_seen.get(fp, 0.0) < SUPPRESS_SECONDS:
        return False  # same workload, same alert, still inside the window
    _last_seen[fp] = now
    return True


# In-memory state dies with the process. A restart mid-rollout lets one
# duplicate through; persist the map if that costs you more than it saves.

That deduplication is the line between 40 raw alerts and 12 investigations, and therefore between a bill you ignore and one you watch. The article puts the cost at about 0.04 dollars per investigation, roughly 12 dollars a month, and I take both as the team's own figures rather than something I measured.

Reproduce the first step on your own Alertmanager

You do not need their playbook to see Holmes reason over a real alert. The Holmes walkthrough port-forwards Alertmanager, optionally posts a test KubePodCrashLooping alert, and runs holmes investigate alertmanager with filters. The skills config and Slack token below are copied from the skills and operator pages.

# 1. Reach Alertmanager (the Holmes walkthrough does it this way)
kubectl port-forward svc/<Your-Alertmanager-Service> 9093:9093

# 2. Investigate every active alert, or narrow it down
holmes investigate alertmanager --alertmanager-url http://localhost:9093
holmes investigate alertmanager --alertmanager-url http://localhost:9093 \
  --alertmanager-alertname "KubePodCrashLooping"
holmes investigate alertmanager --alertmanager-url http://localhost:9093 \
  --alertmanager-label "namespace=production"

# 3. Give Holmes your skills (Holmes 0.26.0 or later), ~/.holmes/config.yaml
cat >> ~/.holmes/config.yaml <<'EOF'
skill_repos:
  - url: https://github.com/<org>/<repo>.git
    branch: main
    sub_path: skills
    token_env: GITHUB_SKILLS_TOKEN
EOF

# 4. In-cluster, post operator results to Slack (Helm values.yaml)
#    additionalEnvVars:
#      - name: SLACK_TOKEN
#        valueFrom:
#          secretKeyRef:
#            name: holmes-secrets
#            key: slack-token

This is the CLI path and the operator path, which are separate. The Slack destination in the operator docs applies to health checks in alert mode and reports sent, failed or skipped per destination. It does not give you the thread-reply behaviour of STCLab's playbook, which is why that glue exists.

Where it broke: model hunting and stale knowledge

The most useful part of the article is the list of failed models, because it shows that tool-calling reliability and infrastructure were the first problems, not reasoning quality.

The team's requirements were Korean output, Slack formatting, runbook retrieval and cross-cluster log correlation, which is why only one model family passed. They ended with a self-hosted model in staging and a managed API in production, switched by a single model-config block with temperature 0, and they swapped the backend three times without changing the pipeline. One upstream fix, a three-line change for pod identity authentication, was merged as PR 1850.

One more limit on the evidence: this is a two-person team's account on their own clusters, in Korean, with a custom playbook. Nothing here shows the same numbers on your alerts, which is exactly why step one on Monday is measuring your own baseline.

What to do on Monday

Do these in order. Each one is small, and the first tells you whether the rest are worth doing.

1. Run one real alert through holmes investigate and count the tool calls that touch a system you do not run. That count is your own version of 16.
2. Write one skill for your noisiest alert with a specific description, a do-not-use list, a stop condition and an owner. Keep the secrets out.
3. Put skills in a Git repo, point skill_repos at it, and add a last_updated field with a quarterly review so stale knowledge gets deleted.
4. Set TOOL_MEMORY_LIMIT_MB deliberately and raise the pod memory limit to match, then add workload-level deduplication before anything posts to a channel.
5. Keep the model in one config block and rerun the same alerts when you swap it, so a model change is a diff in results rather than a new pipeline.

# Step 1: find your wasted tool calls before writing a single skill.
# Run one real alert and count the tool calls Holmes makes that touch a
# system you do not run. Each one is a line for the "do not use" section.
holmes investigate alertmanager --alertmanager-url http://localhost:9093 \
  --alertmanager-alertname "KubePodCrashLooping"

# Step 4: cap tool subprocess memory, and raise the pod limit to match.
export TOOL_MEMORY_LIMIT_MB=2000   # default is 800 MB on x86_64

The article's next step, feeding eBPF network metrics such as TCP retransmits and connection latency from Inspektor Gadget into Prometheus, is the right kind of extension: more evidence for the agent, with the same runbook discipline around it.

The rule I take from this: before choosing a model for an alert-triage agent, write down what is not in your cluster and what a known failure looks like. STCLab's runbooks cost a few markdown files and, in the team's own account, mattered more than the model swaps. Treat the knowledge as code, review it, and let the model stay replaceable.