Alerts
An alert rule watches a number and tells you when it crosses a line. Too many server errors on a route, a slow backend, a gateway that stopped answering: each is a rule. Alerts, under Alerts in the menu, holds the rules of the environment, their history, and where notifications go. A bell in the top bar shows the alerts firing now across the environments whose Alerts you may read.
Who may open it
Reading the rules, the history, the channels and the mail relay needs the view permission of Alerts in the environment (alerts.view). Every change needs the manage permission of Alerts (alerts.manage), which covers creating, changing, muting, switching on and deleting rules, the channels, the mail relay and every test send. The Environment administrator holds both. The Editor and the Viewer hold neither, and see a page that names their role when they open Alerts by its address. A custom role can carry either; see Custom roles.
The bell lists the firing alerts of every environment where you hold the view permission of Alerts, and of no other. Each change on the page is written to the environment's audit log, test sends included.
Traffic rules
A traffic rule reads the numbers Analytics reads, from the environment's Prometheus. Without Prometheus the page says: Prometheus is not connected; only system alerts can fire.
Select New rule on the Rules tab and choose:
- The signal: Response share, the share of answers with a status code; Response time, the p95 response time; or Request count, requests above or below a number.
- The scope: the whole environment, or one route, service, upstream or consumer.
- The window: 5 min, 15 min or 60 min. The condition is read over that window.
- The hold: how long the condition has to hold before the rule fires, and how long it has to clear before the rule resolves. Without a hold the rule fires at the first reading over the line.
An environment holds up to 100 rules.
What a traffic rule measures
The server evaluates every enabled rule once a minute, over the window that ends at that minute.
| Signal | What is read over the window | Fires when |
|---|---|---|
| Response share | The answers with a status class, such as 5xx, or one exact code, as a share of all requests in the scope. | The share is above the threshold, a percentage up to 100. |
| Response time | The p95 response time, either As the client sees it or In the backend. | The p95 is above the threshold, in milliseconds, up to ten minutes. |
| Request count | The number of requests in the scope. | The number is Above or Below the threshold, a whole number. The threshold of a rule for below is at least 1. |
A scope of one upstream counts the requests of every route that reaches it. A rule reads No data rather than firing when the window holds no request at all. A response time rule also reads it when too few requests reached what it measures for a p95. When the p95 lies above the highest response-time bucket, only its lower bound is known. Such a rule fires once that bound is over the threshold, and reads no data until then.
When a rule on a route, a service, an upstream or a consumer reads no requests, the page says why when the gateway's setup explains it. There are three causes, and Where the numbers come from explains each.
- The route is not counted. No copy of the prometheus plugin runs for it. A copy is the plugin on the global rule, or one set on the route itself, its service or its plugin config.
- Its counts are labelled by name. The copy that runs for the route has
prefer_nameon, so its counts carry the route's name instead of its id, and a rule that reads by id finds none. - The label is dropped. The Environment-wide settings of the prometheus plugin can leave labels off every count with
disabled_labels. A rule whose scope needs a dropped label reads nothing.
System alerts
System alerts need no rule of your own. Switch each on from the System alerts card. Each has a fixed hold, and its notification names what it found, up to 25 routes or nodes. The console calls the gateway the engine, as in the first alert's name.
| Alert | Fires when | What the server checks | Hold |
|---|---|---|---|
| Engine unreachable | The gateway's Admin API stops answering G8's checks. | A check of the Admin API with the stored admin key. | 5 minutes |
| Requests not counted | The requests of a route are not counted, so Analytics and the traffic rules miss them. | Whether a copy of the prometheus plugin runs for every route. A route without one is named. | 5 minutes |
| Prometheus unreachable | Prometheus cannot be read. | A read of the environment's Prometheus. | 5 minutes |
| Request records cleanup failed | The daily deletion of old request records failed. | The latest daily cleanup of request records, read from the audit log. | None |
| Gateway node not reporting | A gateway node stopped reporting. | Each gateway node's latest scrape in Prometheus. A node whose last scrape failed is named. | 5 minutes |
| Gateway node configuration behind | A gateway node runs a configuration behind the other nodes. | The configuration revision each node reports to Prometheus. A node behind the highest revision is named. | 15 minutes |
Three of them need no Prometheus: Engine unreachable, Requests not counted and Request records cleanup failed. The other three read it.
States and history
A rule is OK, Pending while the condition holds but the hold has not passed, Firing, or No data when Prometheus has nothing for the window. A rule can be muted for an hour, four hours, a day or a week. No notification is sent for a new firing while the mute lasts. A firing whose notification was already sent still gets one when it resolves.
The History tab lists episodes: when a rule started firing and when it resolved. Episodes are kept for 90 days after they end.
Where notifications go
Notifications are set per environment, on the Settings tab.
- Mail lists of one or more addresses. Mail goes through the environment's Mail relay, which has to offer STARTTLS on the port you choose; 587 is the usual one. An account on the relay is optional.
- Webhooks: a JSON message posted to an address of yours, optionally signed with a shared secret so the receiver can check it came from G8.
Both have a test button that sends a sample.
Test a channel
Every channel row on the Settings tab has Send test. It sends one notification to that channel now, with the status test and the rule name Test notification. It is tied to no rule. The page then says whether the receiver accepted it and how long it took, or why it did not. A mail list whose relay refused some addresses shows which ones.
The mail relay has its own Send test mail to one address you type. The dialog says: One test mail goes through the relay now; nothing is saved. While you edit the relay, the test uses the values in the form, so a new relay can be tried before it is saved.
Each test is written to the audit log with its result. A channel's test leaves an Alert channel tested row (alert_channel.tested), and the relay's test a Mail relay for alerts tested row (alert_smtp.tested).
To see what a notification looks like without waiting for one, open an example from the same tab. The Mail lists card has Preview message, and the Webhooks card has Example request. Each shows a notification the server writes for a 5xx share rule on a route that fires now. The webhook example also shows the headers, signed with an example key.
What a notification carries
A notification is sent when a rule fires and when it resolves. It is also sent when a firing rule is changed, disabled or deleted, as closed, to the channels that got its firing notification.
Mail. The subject reads [g8], then the status, the environment's name and the rule's name. The status is FIRING, RESOLVED, CLOSED or TEST. The body has one line per fact: the rule, the status, the environment, the scope, a summary, the condition and the value read. Where they apply, the detail, the cause and the affected routes or nodes follow, with the reason it closed. The body ends with when the episode started and ended, its number and the delivery ID.
Webhook. A JSON body sent with POST, whose schema is g8.alert.v1. It carries the status (fired, resolved, closed or test), the environment, the rule, the scope, the condition and the value. It also carries the subjects, the started_at and ended_at times, the episode_id and a delivery_id. The same delivery ID is sent in the X-G8-Delivery header, the status in X-G8-Status and the send time in X-G8-Timestamp. When the webhook holds a signing key, the hint of its form explains the signature: Stored sealed. Each notification then carries X-G8-Timestamp and X-G8-Signature: v1= and the hex HMAC-SHA256 of the timestamp, a dot and the body.
The receiver has 15 seconds to answer. Any answer outside 2xx counts as a failure, a redirect included.
When a notification fails
A notification that fails is tried again after 30 seconds, then after 2, 10 and 30 minutes. After the last try it is marked failed. A notification whose channel was deleted is not tried again. Each channel, webhook or mail list, is tried on its own, so a failing receiver does not hold back the others.
The page of an episode on the History tab lists each notification with its channel, its result, its attempts and, while it waits, the time of the next attempt.
Related
- Analytics: the numbers the traffic rules read.
- Environments: the Prometheus connection the rules need.
- Request records: the daily cleanup that the cleanup alert watches.
- Custom roles: a role with the Alerts permissions.