What to watch
The G8 server tells you how it is doing in two ways: three addresses it answers without a session, and a log it writes as one JSON object per line on its standard output. It publishes no metrics of its own. This page lists the addresses, what each answer means, and the log lines your log tooling should raise an alert on.
Who may read them
Anyone who can reach the server's address reads the three addresses; they are public and need no session and no role. The log goes wherever your platform sends a container's standard output, and reading it is a matter of that platform, not of a role in the console.
The three addresses
| Address | What it answers | When it does not |
|---|---|---|
/healthz | A JSON object with the status ok, as soon as the server listens. It checks nothing behind the server. | Only when the process is not listening at all. |
/readyz | A JSON object with the status ready, after two checks within three seconds: the database answers, and every sealed row opens with the keys the process holds. | 503 with the status db unreachable when the database does not answer, or sealed secret needs a key this instance does not hold when a row was sealed under a key the process has not. The pod also logs the row. |
/version | A JSON object with the version of the build, its git_commit and the go_version it was built with. | Never; it reads the binary. |
The chart's Deployment uses the first as its liveness probe and the second as its readiness probe, so a pod that cannot open a sealed row leaves the Service on its own. Rotate the secrets key is the moment the second check matters most; Upgrade G8 is when the third is read.
The log
Every line is a JSON object with the message and its fields. The lines below are worth a rule in your log tooling. Each message is quoted as the server writes it.
At start
| Message | Level | What it says |
|---|---|---|
G8_ variables this server does not read; check them for a misspelling | Warn | A variable under the server's prefix that it does not know, listed under names. A misspelled setting is found here, because the server starts without it; see Server configuration. |
schema ready | Info | The schema migrations ran, with version_from, version_to and applied. Every start prints it; after an upgrade applied names what changed. |
sealed secrets moved to the current key | Info | Rows sealed under the previous key were rewritten under the current one, with the count under rows. Expected once, during a rotation; unexpected at any other time. |
environment seeded from G8_ENVIRONMENT and G8_ENGINE_ADMIN_URL; the table is the source of truth from now on | Info | The first start created the first environment from the settings. It never appears again on this database. |
credentials.view and credentials.manage were retired by migration 012; these custom roles held one of them and hold no consumers.manage, so they no longer read or change a credential: grant consumers.manage to the ones that were meant to | Warn | A custom role lost a retired permission in an upgrade; the roles are listed under roles, and a Platform administrator gives them the new one in Custom roles. |
g8 has no users yet: open the console and create the first administrator with this setup token; every replica of this install prints the same one | Warn | No user exists, and the setup token is printed under setup_token. Expected at the first start; at any later start it means the users are gone, so stop and check the database before anyone uses the token. |
listening | Info | The server accepts connections, with addr and whether tls is on. The line that ends every start. |
server exited | Error | The process stops, with the error that stopped it. A start that fails prints this and nothing after it. |
A start that stops before listening prints server exited with the reason. Two reasons are worth knowing by their text: sealed with a key this server does not hold, with the table and row named, which Rotate the secrets key explains, and G8_TLS_CERT_FILE and G8_TLS_KEY_FILE name a pair that was never opened; nothing is served when the certificate files are set but could not be read.
While it runs
| Message | Level | What it says |
|---|---|---|
not ready: a sealed row needs a key this instance does not hold | Error | The readiness check failed on a sealed row, which is named under error. The pod answers 503 until it holds the key. |
dropped an X-Forwarded-Proto claim from a peer that is not in G8_TRUSTED_PROXIES; list the proxy there if it terminates TLS in front of this server | Warn | A browser request failed the same-site check and carried a forwarded-protocol header from an address not listed as a trusted proxy; the request is refused as cross-site. Behind a proxy that terminates TLS, the console's requests are refused this way until the proxy is listed; see Server configuration. |
audit access.denied | Error | A refused request could not be recorded in the audit log, with the route and the error. The audit log is missing a row it should have. |
audit prune, traffic prune, alert history prune, session purge | Error | A periodic cleanup failed, with its error. The first three also print an Info line with the same message when they ran, and one that ends in skipped: another instance holds the lock when a replica left the run to another. |
alert evaluation | Warn | The minute's evaluation of alert rules had failures, beside the number of environments and rules it read. A rule that cannot be read fires nothing. |
alert dispatch | Error | A notification could not be sent, with the error. Alerts says what the console shows for it. |
Every request the server answers is one Info line with the message request: its method, path, status, duration_ms, request_id and, when known, the client ip. The request_id is also sent back to the client in the X-Request-Id header, so a user's report and the log line meet on it. A rate of lines whose status is a server error is the simplest alert to build on this log.
Numbers about the gateway live elsewhere
The server exports no metrics endpoint: nothing in it answers a Prometheus scrape, and the three addresses above are all it offers. What Prometheus holds is the gateway's traffic, counted by the gateway itself and read by the console on the Analytics page. Where the numbers come from explains how the counts are made and labelled per environment.
The console also raises alerts about an environment's gateway, its Prometheus and its gateway nodes, without a rule of your own. They cover the Admin API or Prometheus unreachable, a route not counted, the daily cleanup of request records failed, and a node not reporting or running an older configuration. Those are the system alerts, switched on per environment and sent to the channels of that environment. Alerts lists them with what each checks and how long it holds.
Related
- Upgrade G8: the lines and addresses an upgrade is checked with.
- Rotate the secrets key: the one time
sealed secrets moved to the current keyis expected. - Back up and restore: the checks a restore ends with.
- Server configuration: the settings the warnings above name.