Alerts

A rule is:

more than N violations a minute on this operation, sustained for M minutes

out through a webhook or a Slack incoming webhook. There is no query language, no expression evaluator and no severity taxonomy, because the only interesting judgement in an alert is the threshold and that one is yours.

The window is the point

A thirty-second spike on a five-minute rule notifies nobody. That is the entire reason the window exists: an alert that fires on every blip is an alert people turn off, and an alert people turn off is worse than none because it is also on the runbook.

A rule fires once when the condition has held for its window, and once more when it stops holding. It does not re-notify every minute in between.

A new rule cannot fire on old news

The first evaluation of a rule establishes a baseline and decides nothing.

Without that, a rule created while an API had been failing all week would fire immediately, as though the week's violations had just happened - and the first thing anybody would learn about our alerting is that it lies. Silencing and re-enabling a rule takes a fresh baseline for the same reason: the gap while it was quiet is not a rate.

How the rate is measured

Violations are stored folded - one row per shape of problem with a cumulative count - so there is no per-minute history to query, and keeping one for every fingerprint on the site would be a large price for a small feature.

Each rule remembers its previous sample instead, and the rate is the difference between two evaluations. An alert's resolution is therefore the period of the evaluator, which runs every minute on the hosted service - one occurrence across the whole fleet, not one per machine. The notification says the rate it measured, so you are never guessing which it was.

Delivery

Slack targets get { "text": ... }. Webhooks get the event - rate, threshold, window, operation, and the same sentence - because a webhook is consumed by a program and a program wants the numbers.

A webhook that is down does not stall anything and is not hammered: the failure is shown on the rule and the next run is the retry. Firing is recorded before the notification is attempted, so a delivery that times out cannot produce the same alert every minute until it comes back.

Target URLs must be public. We POST to them on your behalf, which is the same reason spec URLs have the same rule.

Silencing

One click, and the threshold is kept. A rule that is wrong at 3am should be one click from quiet and one click back, without losing the number somebody spent a week tuning.

From a terminal

apip alert 42
apip alert add 42 "orders drifting" \
    --threshold 10 --window 5 \
    --channel slack --target https://hooks.slack.com/services/...
apip alert silence 42 7
apip alert rm 42 7