Grafana Assistant Investigations: Alert-Triggered RCA Setup

The page arrives at 03:12 with a service name, a severity label and a link to a dashboard. Everything useful still has to be found by a human who is half awake: which deploy, which dependency, which query. That first ten minutes of context gathering is the work Grafana wants Assistant Investigations to take over.

I read Grafana's general-availability entry, the launch press release, the Investigations documentation, the alert enrichment documentation, the IRM webhook guide and the Terraform provider docs for the enrichment resource. I have not run this against a live stack, so every behaviour below is as documented, and where the documentation is silent I say so instead of guessing.

What went generally available, and what this post covers

Grafana's press release lists six tools that reached general availability on 27 July 2026: Assistant Investigations, Assistant Workspace, Assistant Automations, the Grafana Cloud MCP server, gcx and Agent Observability. Investigations is the one that takes an alert or incident, proposes possible causes, checks each against telemetry, and confirms or rules it out. The GA entry says it works across metrics, logs, traces and profiles and ends with a structured report of findings, evidence and suggested next steps.

The release also quotes Grafana's 2026 Observability Survey: 92 percent of practitioners say they would get real value from AI catching anomalies, while only 57 percent say they are implementing observability for their own AI systems in any capacity. This post ignores that gap and covers one narrow, practical question: how to make a report exist before the on-call engineer opens the page.

Four ways an investigation starts, and where each one lands

The GA entry names three ways to start: manually in Assistant Workspace Investigation mode, from an alert enrichment, and from IRM webhooks. The webhook guide splits the third into alert groups and incidents, which behave differently enough that I treat them as separate paths. The table is the whole design space; the choice is mostly about where you want the result to appear.

The practical difference: the enricher puts a link into the notification itself, so it reaches whichever contact point you use, while the webhooks write back into IRM, which is where a team that already runs on-call from IRM will look. The documentation I read does not say whether an enricher investigation and a webhook investigation for the same alert are merged. It documents de-duplication only for repeated alert-group webhook firings, so I would pick one path per alert rather than find out in production.

Path one: an Assistant Investigations enricher, scoped on purpose

Alert enrichment adds context to a notification before it is sent, and the docs list the enrichers: Data Source Query, Assign, External, RCA Workbench, Sift, Explain and Assistant Investigations. The Terraform provider documents the same step as assistant_investigations, and the GA note on alert enrichment confirms the feature itself is generally available in Grafana Cloud. The config below is adapted from the provider's example; the matchers and the Loki step are my choices, not Grafana's recommendation.

# Adapted from the example in the Grafana Terraform provider docs for
# grafana_apps_alertenrichment_alertenrichment_v1beta1.
# The point: scope the investigation step with matchers, never leave it global.
resource "grafana_apps_alertenrichment_alertenrichment_v1beta1" "investigate_payments" {
  metadata {
    uid = "investigate-payments-critical"
  }

  spec {
    title       = "Start an Assistant investigation for critical payments alerts"
    description = "Scoped on purpose: Assistant Investigations has its own pricing"

    # Empty alert_rule_uids and receivers mean "all". The label matchers are
    # what keep this from firing on every alert in the stack.
    label_matchers = [
      {
        type  = "="
        name  = "severity"
        value = "critical"
      },
      {
        type  = "=~"
        name  = "team"
        value = "payments|checkout"
      }
    ]

    # Cheap, deterministic context first: a Loki preview of recent errors.
    step {
      data_source {
        timeout = "30s"
        logs_query {
          data_source_type = "loki"
          data_source_uid  = "REPLACE_WITH_YOUR_LOKI_UID"
          expr             = "{app=\"checkout\"} |= \"error\""
          max_lines        = 5
        }
      }
    }

    # Then the investigation. The enricher adds a link to the notification;
    # the report itself is built in the Assistant, not in the page.
    step {
      assistant_investigations {
        timeout = "30s" # my choice: the docs give no recommended value
      }
    }
  }
}

Each step takes an optional timeout, which the enrichment docs describe as the maximum time it can run before the notification is sent. The docs give no default or recommended value, so the 30s above is mine. Putting a cheap, deterministic Data Source Query step before the investigation means the page still carries a log preview even if the investigation step is slow.

Matchers support the operators equals, not equals, regex match and regex not match, on labels and on annotations. Empty alert_rule_uids and an empty receivers list mean every rule and every receiver, so a spec with neither matcher nor UIDs is the all-alerts scope the warning is about. For a standard user, the alternative is the Alert enrichment tab on a single rule, which needs the write permission and cannot change scope.

Path two: IRM webhooks that write findings back to the page

The recommended setup is in the Assistant, under Settings then Investigations: Enable incident webhooks for the Incident declared trigger, or Add webhook and pick one or more IRM integrations for alert groups. The same screen reports each alert source as Not covered, Configured but disabled or Covered, so you can see what is wired before you assume it. If you configure it by hand in IRM, these are the values the guide gives.

# Manual outgoing webhook in IRM (Integrations > Outgoing Webhooks > Create).
# Use the "Grafana Assistant for IRM" preset when you can; it fills these in.
# Replace STACK_URL with your Grafana Cloud stack URL.

Method:        POST

# Alert groups  (triggers: Alert group created + Status change)
URL:           STACK_URL/api/plugins/grafana-assistant-app/resources/api/v1/investigations/from-irm

# Incidents     (trigger: Incident declared; add Incident changed to keep context fresh)
URL:           STACK_URL/api/plugins/grafana-assistant-app/resources/api/v1/investigations/from-irm-incident

# Credential: a service credential from the SAME stack, not a user session.
# A service account needs this permission to start AND update investigations:
#   grafana-assistant-app.investigations:create
# A 403 from this endpoint usually means that permission is missing.

The credential is where this tends to break. Automatic deliveries need a service credential rather than a user session, and a service account needs the investigations:create permission both to start and to update an investigation. The full setup also needs grafana-irm-app.outgoing-webhooks read and write, integrations read, alert-groups read and write, and plugin access to the IRM app.

Once it fires, an alert group webhook creates an investigation from the alert group title and context, posts a resolution note with a link, and updates the investigation when status or labels change. An incident webhook builds it from the title, summary and available activity, and posts a second activity item with the result. The guide says the webhook payload schema is not in its text, so I use the preset, which sets the endpoint and format itself.

Reading the report: hypotheses are claims, not conclusions

An investigation produces three things you can inspect: the report with findings and next steps, a list of hypotheses with their status and supporting evidence, and the sources, meaning the generated panels and query output with the query, time range and data source behind each. Before a report exists you can disprove or reopen a hypothesis. After it, you answer whether it was the cause. That feedback step is the human check built into the product, and it is the one to use.

It is steerable while it runs. The docs describe a slash command for asking status without changing the investigation, Enter to queue up to five follow-ups, and Cmd or Ctrl plus Enter to redirect immediately. Stop ends evidence gathering and returns a summary marked as incomplete, which you can continue afterwards. Scheduled checks let it watch for recovery.

# Typed into a running investigation in Assistant Workspace.
# /ask reads status without changing the investigation:
/ask which hypotheses are still open?

# Repeat a task at an interval. Checks run at most once a minute,
# up to five per investigation, and stop after 24 hours or 100 checks by default:
/schedule create 10m check whether the storefront error rate has recovered

Customisation is how you stop it rediscovering your architecture every time: infrastructure memory for services and dependencies, rules for standing instructions such as preferred metrics, skills for runbooks it consults when relevant, and MCP servers for systems like code repositories. Reports export as Markdown, HTML or PDF, and snapshots publish the transcript to anyone in your Grafana organisation with the link.

What goes wrong: the failure modes the docs name

I have no outage story to tell here, because I have not run it in production. What I can give you is the failure list from the webhook troubleshooting section and the enrichment warning, mapped from symptom to check. Each row is documented behaviour, not my experience.

A sixth failure is not in the table because it is mine to name: trusting a plausible report. The product exposes hypothesis status and evidence precisely so you can check them, and the cost of skipping that check is acting on a confident wrong cause during an incident.

Trade-offs and guardrails before you automate

Automation has a price and a noise floor. The enricher has its own pricing, investigations count against monthly limits, and every investigation started on a flapping alert is spend with no new information. The webhook guide's own advice is to enable webhooks only for integrations whose alerts you want investigated, start with a few and watch volume, and use automation for common, well-understood alerts while launching manual investigations for complex ones.

The settings give you levers. Under Assistant, Settings, Investigations you can control automatic continuation on alert and IRM updates, continuation windows, start limits, and whether other data sources can be queried. Skipped incident labels take key and value rules such as team:platform: incidents matching a rule do not start investigations, and an investigation already running for a matching incident is cancelled.

The honest trade is latency against load. An investigation that starts at alert time costs money on every qualifying alert and saves minutes on the ones that are real. I would only automate where the alert is frequent enough to have a known shape and severe enough that minutes matter, and leave the long tail manual.

What to do on Monday

Start narrow and make the first week a measurement exercise, not a rollout. This order keeps the blast radius small and gives you evidence before you widen it.

1. In Assistant, Settings, Investigations, search for whether your alert source is covered, and note which sources are Not covered or Configured but disabled.
2. Create a service credential in the same stack with the investigations:create permission, and confirm the people reading reports hold the Assistant Investigation User role.
3. Enable an alert group webhook for one IRM integration only, then watch execution history in IRM OnCall for the first deliveries.
4. If you also want links in notifications, add the enricher through Terraform with label matchers for one team's critical alerts, and never the all alerts scope.
5. After a week, open each report, mark hypotheses as cause or not, compare them with what the responder actually found, and only then widen the scope.

Keep a simple tally during that week: investigations started, investigations a responder used, and wrong top hypotheses. Grafana gives no accuracy figure in the sources I read, so your own tally is the only evidence you will have about your own stack.

The rule I would carry: automate the trigger, never the trust. Pick one start path per alert, scope the enricher with label matchers, give the webhook a proper service credential, and treat every hypothesis as a claim until its evidence has been opened.