Get a Push Notification When a systemd Service Dies

A service dies at 03:40. systemd notices immediately, writes it to the journal, and if Restart=on-failure is set, brings it back. If the restart also fails it gives up, marks the unit failed, and moves on.

All of that is correct behaviour and none of it reaches you. The unit sits in a failed state until something makes you run systemctl status, which is usually a person telling you the site is broken.

OnFailure= is the fix, and it has been in systemd the whole time.

Why OnFailure rather than a cron check

You could poll systemctl is-failed every minute from cron. That works, and it is what most people build first.

OnFailure= is better on both ends. It fires the instant the unit enters a failed state, not up to a minute later, and it fires exactly once per failure rather than every minute the unit stays down. You get the alert sooner and you get one of it.

Step 1: The notify script

Create an application in TheNotificationApp, call it Servers, and copy the app_key. Then save this as /usr/local/bin/notify-failure:

#!/usr/bin/env bash
# notify-failure 
set -uo pipefail

UNIT="${1:?usage: notify-failure }"
APP_KEY="your_app_key_here"
HOST="$(hostname -s)"

# What actually happened, and the tail of why.
RESULT="$(systemctl show -p Result --value "$UNIT" 2>/dev/null)"
STATUS="$(systemctl show -p ExecMainStatus --value "$UNIT" 2>/dev/null)"
LOGS="$(journalctl -u "$UNIT" -n 6 --no-pager -o cat 2>/dev/null | tr '\n' ' ')"

BODY="result=${RESULT:-unknown} exit=${STATUS:-?} :: ${LOGS:-no log lines}"

curl -fsS --max-time 10 -X POST https://thenotification.app/api/sendNotification \
  -H "app_key: $APP_KEY" \
  -H "Content-Type: application/json" \
  -d "$(jq -n --arg t "$UNIT failed on $HOST" --arg b "${BODY:0:180}" \
        '{title: $t, body: $b}')" \
  >/dev/null || logger -t notify-failure "could not send notification for $UNIT"

chmod +x /usr/local/bin/notify-failure

The log tail is what separates this from a useless alert. "api.service failed" makes you SSH in. "api.service failed, exit=137 :: Killed" tells you it was the OOM killer and you can decide from the lock screen whether that needs you now.

Note ${BODY:0:180}: iOS truncates around there anyway, and sending a 4KB body just to have it cut off wastes nothing but reads worse.

Step 2: A templated unit

Write one unit that can report on any other. The @ makes it a template, and %i is whatever follows the @ when it is invoked:

# /etc/systemd/system/notify@.service
[Unit]
Description=Push a notification about %i failing

[Service]
Type=oneshot
ExecStart=/usr/local/bin/notify-failure %i

systemctl daemon-reload

One file, and it now covers every service on the box.

Step 3: Attach it

Do not edit the shipped unit file. Package upgrades overwrite those. Use a drop-in:

systemctl edit api.service

Add:

[Unit]
OnFailure=notify@%n.service

%n expands to the full unit name, so this exact snippet works unchanged on every service you paste it into. That is the whole reason for the template: one drop-in text, any number of units.

systemctl daemon-reload

Step 4: Prove it works

Do not wait for a real failure to find out you had a typo. Make one on purpose:

# /etc/systemd/system/failtest.service
[Unit]
Description=Deliberately fail
OnFailure=notify@%n.service

[Service]
Type=oneshot
ExecStart=/bin/false

systemctl daemon-reload
systemctl start failtest.service

Your phone should buzz with failtest.service failed on hostname. Then delete the unit.

Covering timers too

A systemd timer that runs a failing job leaves the service unit failed, so the same drop-in works on the service half of a timer pair. This is the systemd-native version of wrapping a cron job, and it is strictly better because it captures the exit code and the journal for you:

systemctl edit backup.service   # the service the timer starts

If you are still on cron rather than timers, the equivalent wrapper approach is in getting notified when a cron job fails.

One notification per failure, not per unit

Once this is on a dozen services, a single underlying problem starts sending several notifications. The database goes down, and the API, the worker and the scheduler all fail within twenty seconds of each other. Three buzzes, one cause.

A short debounce in the notify script fixes most of it, using nothing but a timestamp file:

# near the top of notify-failure, after UNIT is set
WINDOW=60
STAMP="/var/tmp/notify-failure.last"

now="$(date +%s)"
last="$(cat "$STAMP" 2>/dev/null || echo 0)"

if [ "$(( now - last ))" -lt "$WINDOW" ]; then
  logger -t notify-failure "suppressed $UNIT, within ${WINDOW}s of the last alert"
  exit 0
fi
echo "$now" > "$STAMP"

The first failure notifies, anything in the following minute is logged and dropped. You still get told, and the journal still has the full picture when you go looking. Sixty seconds is a reasonable default: long enough to collapse a cascade, short enough that two genuinely unrelated failures an hour apart both reach you.

If you would rather never suppress anything, skip this and set the per-unit rate limit below instead. It is a real tradeoff, not an obvious win.

The restart loop problem

One case to know about. With Restart=always and a short RestartSec, a unit that crashes on startup can enter a fast restart loop, and depending on your configuration that can fire OnFailure repeatedly.

Rate limiting is already there for exactly this, so set it deliberately:

[Unit]
StartLimitIntervalSec=300
StartLimitBurst=3

Three failures in five minutes and systemd stops trying, which caps the notifications at three rather than three hundred. Worth setting before you attach this to anything with Restart=always.

The honest part

The free tier is 100 notifications for the lifetime of the account, not per month. For failure-only alerting that is generous, because a healthy month sends zero. The restart loop above is the one realistic way to burn it, which is why the rate limit is in this post rather than a footnote.

Pro is $2.99 a month for 1,000. And jq is not installed by default on a minimal server: apt install jq first, or the script fails silently at exactly the wrong moment.

Where this fits

This is the same single POST as everything else, triggered by systemd instead of by your code. The request shape is in the API reference, and the shell patterns are in sending push notifications from a Bash script.

Grab a key at thenotification.app and let the box tell you when something on it stopped.

New to this? Start with the server-down alert this is built on.