Where the numbers come from
The numbers on Analytics are counts and times the gateway keeps itself. G8 does not count your traffic and does not add up request records. It asks Prometheus, a database for numbers over time, what the gateway's counters say. This page explains how a request becomes a number there, and why a number can differ from what you expected.
- The gateway handles a request and adds one to a counter for that route and status code. It also notes how long the request took.
- On a fixed schedule, Prometheus fetches the counters from each gateway node and stores them with the time, under the label of the environment.
- When you open Analytics, G8's server asks Prometheus how much each counter rose in each time slice of the window, and the console draws the answer.
The gateway counts
The counting is done by the prometheus plugin on the gateway, on a global rule so that it runs on every request of the environment. Whoever installs the gateway writes that rule, as the installation guides show, and G8 only reads it. The global rule of the request records, g8-request-records, is a different one, which G8 writes itself when logging is turned on. A route can also run its own copy of the prometheus plugin, from the route, its service or its plugin config.
A counter is a number that only goes up. The gateway keeps one for every combination of what it counts: the route, its service, the consumer, the status code, the backend node, and who answered. A request to the orders route that one backend node answered with 200 adds one to exactly one counter.
The route in a counter is the route's id, with the route's name in a label beside it. Analytics shows each route under the name the environment's gateway holds for that id now. When no route has that id any more, its row says so: No route has this id in the gateway: it was deleted, or its counts carry its name instead of its id.
Beside the counts, the gateway sorts each request's time into buckets, ranges of milliseconds, and counts the requests in each bucket. It does so three times: the whole time as the client sees it, the time in the backend, and the gateway's own time.
Prometheus keeps the numbers
Prometheus reads the gateway nodes on a schedule, a scrape, and keeps every value it read with the time it read it. The scrape set up for an environment's nodes adds the environment's label to every value, g8_environment="dev" for the environment whose slug is dev. Several environments can share one Prometheus this way.
G8's server reads Prometheus for one environment at a time. Every question it sends names the environment's label, so a count without that label is never read.
How G8 counts
A counter's value is a running total, so G8 never shows it as it is. It splits the window into time slices, at most 96 of them, each from one minute up to eight hours long. For each time slice it asks how much the counter rose.
The first value Prometheus sees of a new counter is where the counter starts, not a rise. The requests already in it when it first appears are not counted. A route's first 404, for example, makes a new counter that Prometheus first reads at one. Analytics does not count that 404, and it counts every 404 after it.
Prometheus can count a time slice exactly or estimate it from the values nearest its edges. Counting exactly takes the anchored range selector, a way of asking Prometheus for a counter's rise from the exact start to the exact end of the time slice. Some Prometheus versions offer it only when started with an extra option, which the installation guides set. G8 asks for the exact count first, and falls back to the estimate when Prometheus does not offer it. The chart then says its counts are estimates, which can gain or lose a few requests at the edges of each time slice.
Response times are read the same way, from the rise of each bucket's count in each time slice. A percentile, p95 for example, is an estimate that falls within one bucket, and the charts say they are estimated from histogram buckets.
Who answered
Every count carries who gave the answer, and whether the gateway had handed the request to a backend node. Analytics shows the answers of the window under By source, with these labels.
| Label | What it means |
|---|---|
| Backend | The backend sent the answer. |
| Unreachable | Handed to a node, and no answer came back: refused, timed out, broken off, or given up once the retries ran out. Shown among server errors. The 504 the gateway gives when the chosen node did not answer in time counts here, because no node answered. |
| Client closed | Handed to a node, and no answer reached the client, most often because it closed the connection first (499). Shown among client errors. |
| Gateway | The gateway answered before it handed the request to any node. A refused key or a limit, for example. |
An upstream's own numbers count only the requests handed to one of its nodes. A request the gateway refused before it chose a node counts on the route and not on the upstream.
Routes that are not counted
A route is counted when the global rule's prometheus plugin is on, or when a copy of the plugin runs on the route. When the global rule's copy is switched off, only the routes that run a copy of their own are counted. G8 reads the routes of the environment and says which ones fall outside: 2 of 12 routes are not counted: no copy of the prometheus plugin runs on them.
The plugin can also be set to label counts with the route's name instead of its id, with prefer_name. Those counts no longer match the route's id, so they read empty on the route: This route labels its counts by name, so they read empty here: prefer_name is on in the copy that runs for it.
Numbers, not requests
Prometheus holds counts, not the requests behind them. To see one request, what it carried and what the backend answered, use the request records; Where request records go explains where they live.
Related
- Analytics: the page that shows these numbers, and what has to be connected.
- Alerts: rules that read the same numbers.
- Where a plugin runs: global rules and the copies of a plugin.