Send Grafana Alerts to Your Phone

Grafana alerting is genuinely good at the hard part. It evaluates the rule, handles the pending period so a single scrape does not trip it, tracks state, and knows the precise moment your error rate crossed the line.

Then it puts that on a dashboard. Which is a strange place to put an alert, because the defining feature of an alert is that you are not looking.

Contact points are the way out, and the webhook one takes about five minutes with no bot token, no SMTP server, and no per-seat pager subscription.

What you need

• Grafana 9 or later, using the unified alerting stack, which is the default
• An app key from TheNotificationApp
• The iOS app installed and signed in with the same Apple ID

Step 1: Get your app key and test it

Create an application in the app, call it Grafana, and copy the app_key. Build the webhook URL:

https://thenotification.app/api/webhook/static/?app_key=YOUR_API_KEY&title=Grafana

Test it from a terminal before you go anywhere near Grafana. Debugging one unfamiliar thing is much easier than debugging two:

curl -X POST "https://thenotification.app/api/webhook/static/?app_key=YOUR_API_KEY&title=Grafana" \
  -H "Content-Type: application/json" \
  -d '{"status":"firing","alert":"HighErrorRate","value":"7.2%"}'

Phone buzzed? Good. The rest is Grafana configuration.

Step 2: Create the contact point

Go to Alerting, then Contact points, then Add contact point.

• Name: iPhone
• Integration: Webhook
• URL: the URL from step 1
• HTTP Method: POST

Leave the optional auth fields empty. The key is already in the query string, which is what makes this work with a plain webhook integration and no custom headers.

Hit Test. Grafana sends its sample payload and your phone should show it, formatted into readable text by the static webhook.

Step 3: Route alerts to it

A contact point does nothing until a notification policy points at it. Go to Alerting, then Notification policies.

The quick version: edit the default policy and set its contact point to iPhone. Every alert now reaches your phone.

The version you actually want once you have more than a handful of rules is a nested policy, so only the alerts that deserve a phone buzz get one. Add a child policy that matches a label:

- matcher: severity = critical
  contact point: iPhone
- matcher: severity = warning
  contact point: Slack

Then label your rules accordingly. Critical goes to the phone, everything else goes somewhere you check on purpose. This is the single decision that determines whether the notifications stay useful in three months.

Step 4: Stop a flapping rule from spending your quota

Grafana groups and repeats notifications, and the defaults are tuned for chat, not for a lock screen. Three settings in the notification policy matter:

• Group by: alertname plus whatever identifies the instance. Without grouping, forty firing series produce forty notifications.
• Group interval: 5 minutes. How long Grafana waits before sending an update about a group that changed.
• Repeat interval: 4 hours, or longer. This is the one that bites. Left at the default, an alert that stays firing re-notifies you on a loop all night.

Set the repeat interval deliberately. An alert that fires at 2am and re-sends every thirty minutes until morning is sixteen notifications for one incident, and it will train you to swipe the channel away.

Making the notification readable

The static webhook formats Grafana's whole payload, which includes labels, annotations, values and a couple of URLs. It is complete, and it is long.

If you want something tighter, use the dynamic webhook and name the fields:

https://thenotification.app/api/webhook/dynamic/?app_key=YOUR_API_KEY&title=title&body=message

Grafana's payload has both title and message at the top level, so that gives you a clean two-line notification. You control both through the rule's summary and description annotations, which means you can write the notification text in the alert rule itself:

Summary:     {{ $labels.instance }} error rate at {{ $value | printf "%.1f" }}%
Description:  Above 5% for 10 minutes. Check the API dashboard.

The tradeoff is the usual one: naming fields couples you to the payload shape, so a Grafana upgrade that renames something breaks the notification quietly. The static webhook never breaks and is just wordier. Details for both are in the dynamic webhook docs.

Adding a tap-through to the panel

The dynamic webhook takes a url parameter naming a field to use as the notification link. Grafana includes dashboard and panel URLs in its payload, so the alert can open the exact graph that fired it:

...&title=title&body=message&url=externalURL

Now a 2am alert is one tap from the chart instead of a hunt through bookmarks while half awake.

The honest part

The free tier is 100 notifications for the lifetime of the account, not per month. Grafana is the tool most likely to blow through that, because a badly grouped alert on a flapping series can send dozens in an hour. Set the grouping and the repeat interval before you route anything real to it.

Pro is $2.99 a month for 1,000, which is realistic for a small setup with sane routing. If you are running enough alerts to need more than that, you are past the point where a phone notification is the right primary channel anyway.

Where this fits

Grafana is worth it when you already have metrics worth graphing. If all you want is "tell me when the box stops answering", ten lines of bash and cron gets you there without the stack: that version is in getting a server down phone notification. The underlying request is documented in the API reference.

Grab a key at thenotification.app and give your alerts somewhere to land.