The Gap Grafana Leaves Open
You've built a solid observability stack in Grafana. Your dashboards are crisp, your alert rules fire reliably, and your notification policies are wired up with label matchers so that a database alert doesn't ping the frontend team. That part works well.
What Grafana doesn't have is on-call schedule management. There's no concept of who is currently on-call for a given service, no rotation logic, no escalation paths if the first responder doesn't acknowledge within five minutes, and no incident timeline to track what happened after the alert fired. Grafana gets the alert to a destination — Slack, PagerDuty, a webhook — but the routing logic beyond that has to live somewhere else.
If you're using a generic webhook contact point today, or paying for PagerDuty on top of Grafana, this post walks you through connecting Grafana to Alert24 so that the alert labels you're already setting drive the on-call routing automatically.
How Grafana Notification Policies Work
Before connecting anything, it helps to understand what Grafana actually sends and how its routing decisions happen.
Grafana's alerting model has three layers:
Alert rules fire based on query thresholds and attach labels. You control those labels directly in the rule definition. Common patterns:
# Labels attached to an alert rule
labels:
team: "payments"
service: "checkout-api"
severity: "critical"
env: "production"
Contact points define where an alert goes — Slack channels, email, PagerDuty, or a generic webhook URL. Each contact point is a delivery mechanism, not a routing decision.
Notification policies are the routing layer. A policy is a tree of matchers. When an alert fires, Grafana walks the policy tree and finds the first node whose label matchers all match, then sends the alert to that node's contact point.
A policy node for the payments team might look like this in Grafana's UI:
- Matcher:
team = payments - Contact point:
Alert24 Payments Webhook - Group by:
service,severity - Repeat interval:
4h
That's the right mental model: Grafana decides which contact point to use based on labels, and your job is to put the right webhook URL on the contact points that map to each team.
Setting Up Alert24 as a Webhook Contact Point
In Alert24, each team has its own webhook ingest URL. Navigate to your team settings, open the Integrations tab, and copy the webhook URL for the team you want to connect first.
In Grafana, go to Alerting > Contact points and create a new contact point:
- Type: Webhook
- URL: the Alert24 webhook URL for that team
- HTTP Method: POST
- Authorization header: If your Alert24 webhook requires a token (recommended), set
Authorization: Bearer <your-token>here
Alert24's webhook receiver accepts Grafana's native alert payload without any transformation. The labels, annotations, and state information from Grafana arrive as-is and are available to Alert24's routing engine.
Test the contact point using Grafana's built-in test button before wiring it into a notification policy. You should see a test incident appear in Alert24 within a few seconds.
Using Alert Rule Labels to Drive Routing
The labels you attach to Grafana alert rules become the routing signal inside Alert24. Alert24 routing rules inspect the incoming webhook payload and match against those labels to decide which on-call schedule to page.
Here's a practical label convention that scales well across multiple teams:
| Label | Example values | Purpose |
|---|---|---|
team |
payments, infra, data |
Maps to an Alert24 team and its on-call schedule |
service |
checkout-api, postgres-primary |
Used in the incident title and for deduplication |
severity |
critical, warning |
Controls escalation urgency in Alert24 |
env |
production, staging |
Filter out staging noise from on-call pages |
In Alert24, create a routing rule for each team that matches the team label:
if payload.labels.team == "payments"
→ route to: Payments On-Call Schedule
For severity-based escalation, you can layer a second condition:
if payload.labels.team == "payments"
AND payload.labels.severity == "critical"
→ route to: Payments On-Call (immediate escalation)
→ escalate to: Payments Manager after 10 minutes
Warning-severity alerts from the same team can route to a lower-urgency policy that only sends a Slack message and doesn't wake anyone up.
A Complete Routing Example
Say you have three teams: infra, payments, and data. Each has its own Grafana notification policy node and its own Alert24 contact point.
In Grafana, your notification policy tree looks like this:
Default policy
├── Matcher: team = infra
│ └── Contact point: Alert24 - Infra
├── Matcher: team = payments
│ └── Contact point: Alert24 - Payments
└── Matcher: team = data
└── Contact point: Alert24 - Data
In Alert24, each webhook contact point routes to the correct team's on-call schedule. The infra team's schedule might rotate weekly among four engineers. The payments team might run a follow-the-sun rotation. Each team manages its own schedule independently without touching Grafana at all.
When a Postgres replication lag alert fires with team=data and severity=critical, the flow is:
- Grafana evaluates the alert rule threshold — breached.
- Grafana walks the notification policy tree, matches
team = data. - Grafana posts the alert payload to the Alert24 - Data webhook.
- Alert24 receives the payload, evaluates routing rules, identifies the on-call engineer for the data team.
- Alert24 pages that engineer via SMS, phone call, or push notification.
- If no acknowledgment within your configured window, Alert24 escalates to the secondary or team lead.
- The incident timeline records every step: fired, paged, acknowledged, resolved.
The Grafana side stays clean. You're not maintaining a list of phone numbers in Grafana, and your on-call rotations don't require any Grafana changes when someone joins or leaves a team.
Filtering Environments at the Notification Policy Level
One practical addition: filter out staging and development alerts before they reach Alert24's on-call routing. Add an env = production matcher as a parent node in your notification policy, or add it as a condition in your Alert24 routing rules.
In Alert24 routing rules:
if payload.labels.team == "payments"
AND payload.labels.env == "production"
AND payload.labels.severity == "critical"
→ page on-call
This prevents a staging deploy gone wrong from waking your payments engineer at 2am. Staging alerts can still route to a separate Slack-only contact point if you want visibility without on-call interruption.
Next Steps
Start with one team. Pick your most critical service, attach consistent labels to its alert rules, create the Alert24 webhook contact point in Grafana, and build the notification policy matcher. Verify end-to-end with a test alert before you go live.
Once the first team is working, the pattern repeats: new team, new labels, new contact point, new routing rule in Alert24. Your Grafana notification policy tree stays maintainable because each branch is a single label matcher, not a sprawling list of Slack channels and phone numbers.
If you don't have an Alert24 account yet, start a free trial — the webhook integration takes about ten minutes to configure and you can bring your own Grafana instance without any changes to your existing alert rules beyond adding the team label.