Get a Push Notification When a GitLab CI Pipeline Fails

GitLab emails you when a pipeline fails. That email arrives in the same folder as the merge request notifications, the comment notifications, and the six other pipelines that passed, and you read it whenever you next open your mail.

Meanwhile main has been red for forty minutes and the deploy that was supposed to go out at four has not.

One extra job in .gitlab-ci.yml fixes it, and GitLab hands you everything the notification needs in predefined variables.

Step 1: Store the key as a CI variable

Create an application in TheNotificationApp, call it GitLab, and copy the app_key.

In your project, go to Settings, then CI/CD, then Variables, then Add variable:

• Key: TNA_APP_KEY
• Value: your app key
• Type: Variable
• Mask variable: on
• Protect variable: on, if you only run this on protected branches

Masking is the setting that matters. Without it, any job that echoes the environment prints your key into a build log that anyone with read access can see, and CI logs live for a long time.

Do not put the key in .gitlab-ci.yml. That file is in the repository, which means the key is in the history forever, including after you rotate it.

Step 2: Add the notify job

stages:
  - build
  - test
  - notify

notify_failure:
  stage: notify
  image: curlimages/curl:latest
  when: on_failure
  variables:
    GIT_STRATEGY: none
  script:
    - |
      curl -fsS -X POST https://thenotification.app/api/sendNotification \
        -H "app_key: $TNA_APP_KEY" \
        -H "Content-Type: application/json" \
        -d "{
          \"title\": \"Pipeline failed: $CI_PROJECT_NAME\",
          \"body\": \"Branch $CI_COMMIT_REF_NAME by $GITLAB_USER_NAME. $CI_COMMIT_TITLE\",
          \"link\": \"$CI_PIPELINE_URL\"
        }"

Three things are carrying their weight here.

when: on_failure means the job only runs if something earlier in the pipeline failed. It is the entire mechanism, and it needs the job to be in a later stage than whatever it is watching.

GIT_STRATEGY: none skips the repository clone. This job does not need your code, and skipping the clone takes the notification from roughly thirty seconds to about two, which matters when the whole point is speed.

The curlimages/curl image is tiny and has curl already. Using your build image here would mean pulling a large image just to send an HTTP request.

$CI_PIPELINE_URL in the link field is what makes the notification useful rather than merely informative: tapping it opens the failed pipeline, so you can triage from your phone.

Step 3: Only notify about branches you care about

As written, every failed pipeline on every feature branch buzzes your phone. That includes the four failures you cause yourself while iterating, which is exactly how people end up muting this.

Narrow it with rules:

notify_failure:
  stage: notify
  image: curlimages/curl:latest
  variables:
    GIT_STRATEGY: none
  rules:
    - if: '$CI_COMMIT_BRANCH == $CI_DEFAULT_BRANCH'
      when: on_failure
    - if: '$CI_COMMIT_TAG'
      when: on_failure
    - when: never
  script:
    - |
      curl -fsS -X POST https://thenotification.app/api/sendNotification \
        -H "app_key: $TNA_APP_KEY" \
        -H "Content-Type: application/json" \
        -d "{
          \"title\": \"Pipeline failed: $CI_PROJECT_NAME\",
          \"body\": \"Branch $CI_COMMIT_REF_NAME by $GITLAB_USER_NAME. $CI_COMMIT_TITLE\",
          \"link\": \"$CI_PIPELINE_URL\"
        }"

Now only the default branch and tags notify. A broken feature branch is your problem and you already know about it, because you are looking at it. A broken main is everyone's problem and nobody is looking.

Watch out for the commit message

The example above interpolates $CI_COMMIT_TITLE straight into a JSON string. A commit message containing a double quote produces invalid JSON and the notification fails, which you will discover on the day someone commits fix "off by one" in parser.

If your team writes free-form commit messages, build the payload with a tool that escapes properly:

  image: alpine:latest
  script:
    - apk add --no-cache curl jq
    - |
      jq -n \
        --arg t "Pipeline failed: $CI_PROJECT_NAME" \
        --arg b "Branch $CI_COMMIT_REF_NAME by $GITLAB_USER_NAME. $CI_COMMIT_TITLE" \
        --arg l "$CI_PIPELINE_URL" \
        '{title: $t, body: $b, link: $l}' \
      | curl -fsS -X POST https://thenotification.app/api/sendNotification \
          -H "app_key: $TNA_APP_KEY" \
          -H "Content-Type: application/json" \
          -d @-

Slightly slower because of the package install, and it cannot be broken by punctuation.

Knowing which stage failed

A pipeline with build, test, and deploy stages fails very differently depending on which one broke. GitLab does not hand you the failed stage directly, but you can approximate it by adding a per-stage notify job with needs, or more simply by naming the job in the title:

notify_deploy_failure:
  stage: notify
  needs: ["deploy"]
  when: on_failure
  script:
    - |
      curl -fsS -X POST https://thenotification.app/api/sendNotification \
        -H "app_key: $TNA_APP_KEY" \
        -H "Content-Type: application/json" \
        -d "{\"title\": \"DEPLOY failed on $CI_COMMIT_REF_NAME\", \"link\": \"$CI_PIPELINE_URL\", \"body\": \"$CI_COMMIT_TITLE\"}"

A failed deploy is worth waking up for. A failed lint job is not, and separating them is what keeps the channel trustworthy.

Reusing it across projects

Copying that job into fifteen .gitlab-ci.yml files means fifteen places to fix when you change it. GitLab has include for exactly this.

Put the job in a shared repository, as notify.yml:

# ci-templates/notify.yml
.notify_failure_template:
  stage: .post
  image: curlimages/curl:latest
  variables:
    GIT_STRATEGY: none
  rules:
    - if: '$CI_COMMIT_BRANCH == $CI_DEFAULT_BRANCH'
      when: on_failure
    - when: never
  script:
    - |
      curl -fsS -X POST https://thenotification.app/api/sendNotification \
        -H "app_key: $TNA_APP_KEY" \
        -H "Content-Type: application/json" \
        -d "{
          \"title\": \"Pipeline failed: $CI_PROJECT_NAME\",
          \"body\": \"Branch $CI_COMMIT_REF_NAME by $GITLAB_USER_NAME\",
          \"link\": \"$CI_PIPELINE_URL\"
        }"

Then in each project:

include:
  - project: 'your-group/ci-templates'
    file: '/notify.yml'

notify_failure:
  extends: .notify_failure_template

Two lines per project, and the TNA_APP_KEY variable can live at the group level rather than being re-entered fifteen times. The .post stage is built into GitLab and always runs last, so you do not need to declare a notify stage at all.

The honest part

The free tier is 100 notifications for the lifetime of the account, not per month. Scoped to the default branch, a healthy project sends a handful a month and that lasts a long time. Left on every branch, an active team can spend it in a week.

Pro is $2.99 a month for 1,000. And this reaches iPhone only, because it rides Apple's push service.

Where this fits

The GitHub Actions version of the same idea is in GitHub push notifications on your iPhone, and the request shape is in the API reference.

Grab a free key at thenotification.app and find out that main is red while you can still do something about it.

New to this? Start with the server-down alert this is built on.