Know When a Scheduled AI Research Run Has Something Worth Reading

Setting up a daily research agent feels great for about four days. Every morning at seven it reads the feeds, the changelogs, the competitor pages, and sends you a summary. By the second week you are swiping the notification away without opening it, because eleven of the last fourteen mornings had nothing in them worth knowing.

The agent is not the problem. It is doing exactly what you asked. The problem is that "the run finished" and "the run found something" got wired to the same alert, and only one of those is worth interrupting you for.

Two different events

A scheduled research run has two outcomes you might care about, and they deserve different treatment:

• It found something. Notify. This is the whole reason the job exists.
• It ran and found nothing. Log it, do not notify. A quiet morning is information, but it is not urgent information.

There is a third one people forget: it did not run at all. That is the case a daily notification accidentally covers, because the absence of your 7am ping is a signal. Once you gate the notification you lose that, so it needs handling separately. More on that below.

Ask the model to score its own output

The gate needs a number, and the cheapest place to get one is the run you are already paying for. Ask for structure instead of prose:

SCHEMA_HINT = """
Return JSON only:
{
  "relevance": 0-10,
  "headline": "one line, under 80 characters",
  "why": "one sentence on why this matters, or why nothing does",
  "sources": ["url", ...]
}

Score honestly. A 3 means "mildly interesting", a 7 means "you would want to
know today", a 9 means "drop what you are doing". Most days are a 2.
"""

That last sentence does more work than the rest of the prompt. Without an explicit anchor, models grade generously and everything comes back a 7, which puts you right back where you started.

The gate

import json, os, requests

THRESHOLD = 6

def notify(title: str, body: str, link: str | None = None) -> None:
    try:
        requests.post(
            "https://thenotification.app/api/sendNotification",
            headers={"app_key": os.environ["TNA_APP_KEY"], "Content-Type": "application/json"},
            json={"title": title, "body": body, **({"link": link} if link else {})},
            timeout=10,
        )
    except Exception as exc:
        print(f"[notify] {exc}")

def report(result: dict) -> None:
    score = int(result.get("relevance", 0))
    headline = result.get("headline", "").strip()

    if score < THRESHOLD:
        print(f"[quiet] scored {score}: {headline}")
        return

    sources = result.get("sources") or []
    notify(
        f"Research: {headline}"[:120],
        f"{result.get('why', '')} ({score}/10, {len(sources)} sources)"[:180],
        link=sources[0] if sources else None,
    )

Everything still gets logged. You keep the full record for the weekly skim, and your phone stays out of it unless the run cleared the bar.

Stop it repeating itself

The next failure is subtler. A genuinely interesting story stays interesting for three days, and a naive gate notifies you about it every one of them.

Hash the headline and remember what you have already sent:

import hashlib, pathlib, json

SEEN = pathlib.Path.home() / ".research-seen.json"

def already_sent(headline: str, keep: int = 60) -> bool:
    key = hashlib.sha256(headline.lower().encode()).hexdigest()[:16]
    history = json.loads(SEEN.read_text()) if SEEN.exists() else []
    if key in history:
        return True
    history.append(key)
    SEEN.write_text(json.dumps(history[-keep:]))
    return False

Exact-match hashing is blunt, and a reworded headline slips through. It still removes most of the repetition for ten lines, and the alternative is embedding similarity, which is a real amount of machinery for a problem this size.

Covering the run that never happened

Now that silence is normal, a broken cron job looks exactly like a quiet Tuesday. Fix that with a heartbeat rather than by weakening the gate: have the job touch a file every time it completes, and check the file's age separately.

#!/usr/bin/env bash
# check-research-heartbeat.sh, run from cron a few hours after the research job
STAMP="$HOME/.research-last-run"
if [ ! -f "$STAMP" ] || [ "$(find "$STAMP" -mmin +1500)" ]; then
  curl -fsS -X POST https://thenotification.app/api/sendNotification \
    -H "app_key: $TNA_APP_KEY" \
    -H "Content-Type: application/json" \
    -d '{"title":"Research job is stale","body":"No completed run in over 25 hours."}' \
    >/dev/null
fi

Two jobs, two responsibilities: one tells you when there is news, the other tells you when the thing that finds news has stopped. That separation is the same shape as getting notified when a cron job fails, which is worth reading if the scheduled side is what you are building.

Tuning the threshold

Start at 6 and adjust once, after two weeks, based on what actually happened:

• Buzzed and you did not open it: raise it.
• Found something in the logs you wish you had known: lower it.
• Nothing either way for a fortnight: the research prompt is the problem, not the gate.

Resist tuning it daily. You need a couple of weeks of runs before the signal means anything.

The honest part

The point of the gate is that it sends far fewer notifications, so the free tier of 100 for the lifetime of the account genuinely lasts. At a couple of real findings a week that is most of a year. Pro is $2.99 a month for 1,000 if you run several of these.

The tradeoff you are making is real, though: a gate set too high means the run finds something and you never hear about it. That is a worse failure than being over-notified, because it is invisible. Read the quiet log occasionally. If the threshold is wrong, that is the only place it shows.

The request shape is in the API reference. Grab a key at thenotification.app and let the boring mornings stay quiet.

New to this? Start with what an MCP server is.