How to Create Trackable Incidents from Grafana Alert Rules

Grafana Alerts Without Incident Management Are Half the Job

Your Grafana alert fires. Maybe it sends a Slack message, maybe an email. Someone sees it, investigates, and fixes the problem. An hour later your manager asks: "When did that start? Who handled it? How long were we down?" You open a Slack thread and start piecing together a timeline from messages.

That's the gap. Grafana is genuinely good at evaluating alert rules and routing notifications. What it does not do is create a structured incident record — something with a start time, an owner, a severity level, a resolution timestamp, and an audit trail your team can actually reference. That's a separate concern, and wiring it up takes about fifteen minutes if you know where the pieces go.

This post walks through configuring a Grafana webhook contact point that creates incidents in Alert24 when an alert fires, resolves them automatically when the alert clears, and uses alias-based deduplication so a flapping alert doesn't generate twenty separate incident records.

How the Grafana Webhook Integration Works

Grafana's alerting system supports webhook contact points. When an alert rule transitions state — from Normal to Firing, or from Firing to Resolved — Grafana POST-s a JSON payload to the URL you specify. Alert24 exposes an incident API that accepts that payload and creates or updates an incident accordingly.

The flow looks like this:

1. Alert rule crosses its threshold and enters Firing state.
2. Grafana evaluates the contact point and sends a webhook POST.
3. Alert24 receives the payload, creates an incident, and assigns it based on your on-call schedule.
4. When the alert resolves, Grafana sends another POST and Alert24 closes the incident, recording the total duration.

The alias field is the key to deduplication. If two webhook calls carry the same alias, Alert24 treats the second call as an update to the existing incident rather than creating a new one. This is essential for Grafana because alert rules can fire repeatedly during a flapping window.

Step 1 — Create the Grafana Message Template

Grafana lets you define message templates that control the shape of the webhook body. Navigate to Alerting > Contact points > Notification templates and create a new template named alert24-incident.

Paste the following:

{{ define "alert24-incident" }}
{
  "alias": "grafana-{{ (index .Alerts 0).Labels.alertname }}-{{ (index .Alerts 0).Labels.instance }}",
  "message": "{{ (index .Alerts 0).Annotations.summary }}",
  "description": "{{ (index .Alerts 0).Annotations.description }}\n\nDashboard: {{ (index .Alerts 0).GeneratorURL }}",
  "severity": "{{ if eq (index .Alerts 0).Labels.severity "critical" }}critical{{ else if eq (index .Alerts 0).Labels.severity "warning" }}medium{{ else }}low{{ end }}",
  "status": "{{ if eq .Status "resolved" }}resolved{{ else }}open{{ end }}",
  "tags": ["grafana", "{{ (index .Alerts 0).Labels.alertname }}"],
  "source": "Grafana"
}
{{ end }}

A few things worth noting here. The alias is constructed from the alert name and the instance label, which gives you a unique but stable identifier across firings of the same rule against the same target. If your alerts don't use an instance label, substitute whatever label makes the alert unique — job, namespace, cluster, whatever fits your label scheme.

The severity mapping translates Grafana's label conventions into Alert24 severity levels. Grafana itself does not enforce a severity label, so your team gets to define the convention. The template above treats critical as-is and maps warning to medium. Adjust those mappings to match how your rules are labeled.

Step 2 — Configure the Webhook Contact Point

In Grafana, go to Alerting > Contact points and add a new contact point. Set the type to Webhook.

The Authorization header is how Alert24 authenticates the request. Generate an API key in your Alert24 account under Settings > API Keys and paste it here. Keep this out of version control — Grafana stores it encrypted in its database.

Enabling Send resolved is what closes the incident automatically. When you enable this, Grafana sends a second webhook call with .Status set to "resolved" when the alert returns to Normal state. The template above maps that to "status": "resolved" in the payload, and Alert24 closes the incident and records the resolution time.

Step 3 — Map Labels to Alert24 Fields

Alert24's incident API accepts a JSON body. Here is what the full payload looks like when a real alert fires — useful to have on hand when you're debugging:

{
  "alias": "grafana-HighMemoryUsage-web-01.prod",
  "message": "High memory usage on web-01.prod",
  "description": "Memory usage exceeded 90% for more than 5 minutes.\n\nDashboard: https://grafana.example.com/d/abc123?...",
  "severity": "critical",
  "status": "open",
  "tags": ["grafana", "HighMemoryUsage"],
  "source": "Grafana"
}

When Alert24 receives this, it checks whether an open incident with this alias already exists. If one does, it updates it. If not, it creates a new incident, timestamps the creation, and triggers your on-call notification policy — SMS, phone call, push notification, or whatever your escalation chain specifies.

When the resolved payload arrives:

{
  "alias": "grafana-HighMemoryUsage-web-01.prod",
  "status": "resolved"
}

Alert24 finds the open incident by alias, closes it, and records the duration. That duration shows up on your incident timeline and in your monthly reports.

Step 4 — Attach the Contact Point to a Notification Policy

Creating the contact point is not enough — you need to route alerts to it. In Alerting > Notification policies, either set the webhook contact point as your default, or add a specific policy that matches the label selectors for alerts you want tracked as incidents.

If you only want production alerts to create incidents, add a matcher like env = prod and point that policy at the Alert24 contact point. Development and staging alerts can continue routing to Slack or email without generating incident records.

Testing the Integration

Before you rely on this in production, trigger a test alert manually. Grafana has a Test button on each contact point that fires a synthetic webhook. Check your Alert24 incident dashboard — you should see a new incident appear within a few seconds with the message, severity, and tags from your template.

If you need to test the full lifecycle, you can also send the resolved payload manually with curl:

curl -X POST https://api.alert24.co/v1/incidents \
  -H "Authorization: Bearer YOUR_API_KEY" \
  -H "Content-Type: application/json" \
  -d '{"alias": "grafana-HighMemoryUsage-web-01.prod", "status": "resolved"}'

This is handy for clearing test incidents and confirming that the resolution path works end to end.

What You Have Now

After this setup, every Grafana alert that fires creates a structured incident record with a start time, severity, on-call assignment, and an automatic close when the alert resolves. Your team gets a timeline of what happened and when, not a scattered Slack thread. Postmortems become easier because the data is already there.

The next steps depend on how your team works. If you want to add an acknowledgment step — so an on-call engineer can claim an incident before it escalates — that is configured in Alert24 under your escalation policy. If you want a public status page that updates automatically when incidents are created or resolved, Alert24 handles that too through the same incident record.

Start with one alert rule that matters. Wire it up, verify the lifecycle works, then expand from there.