Skip to main content

Where request records go

A request record is what the gateway noted about one request: when it came, its method, path and headers, the route that took it, the backend node that answered, the status and how long it took. The gateway writes it into the environment's log store, Elasticsearch or OpenSearch, and G8 reads it from there. This page follows a record from the gateway to the store, back to the console and out again. Request records describes the page that shows them.

  1. Logging is turned on in Traffic Explorer, on its Settings tab, with the Turn on traffic logging dialog. G8 then writes a global rule to the gateway that runs a logger on every request.
  2. The gateway masks the values the masking settings name, then sends the record to the log store in a batch, into the index of the environment and the day.
  3. The log store passes each record through the ingest pipeline G8 ships, which keys it by its request id.
  4. When you open the records, G8's server searches the environment's days of the window.
  5. Once a day, G8 deletes the days older than the environment's retention, one by one, and writes the deletion to the audit log.

The rule G8 writes​

Turning logging on writes one global rule, g8-request-records, which carries elasticsearch-logger. A global rule runs on every request of the environment, so every route is recorded. A stream route is not, because no global rule runs on one. The rule's logger fills in the fields G8 reads, the request id among them.

Before the record leaves the gateway, the logger's Pre-function replaces secret values with ********. Pre-function is one of the execution settings every plugin carries: a function that runs on the gateway before the plugin itself, stored as pre_function. G8 writes its own there. It replaces the values of the request headers, query arguments and JSON body fields that the Masking settings on the Settings tab name.

  • Turning logging on starts the lists with authorization, cookie and x-api-key among the request headers, and token and apikey among the query arguments.
  • Response headers are not masked; they are stored as they arrive.
  • When the function fails on a request, that record is stored without its headers, query arguments and bodies.

The gateway writes with an address and an account of its own. You enter them in the Turn on traffic logging dialog: Log store address, as the gateway reaches it, a Username, and a Password or secret reference. A reference is resolved on the gateway and never passes through g8. This account is separate from the one G8's server reads with, which a Platform administrator enters in the environment's settings. Turning logging on checks that the gateway's account can write records and cannot read them. An account that can read is refused: This account can read records. Give an account that can only write records.

Traffic Explorer has no control that turns logging off. Logging stops when the logger on g8-request-records is switched off, or the rule is deleted, under global rules. With the logger switched off, the Settings tab says: The logger is switched off in the rule. With the rule deleted, the page shows the title Traffic logging is not turned on in place of the records.

One index per UTC day, per environment​

The logger writes one index per UTC day, named after the environment and the day, g8-{environment}-access-{!%Y.%m.%d}. G8 fills {environment} with the environment's slug when it writes the rule, and the gateway fills {!%Y.%m.%d} with the year, month and day in UTC as it writes each record. The day is the day in UTC, whatever the time zone of the gateway or of the reader. For the environment dev, a request at 01:30 on 2 October in Istanbul is still 1 October in UTC, and goes to g8-dev-access-2026.10.01.

Several environments can share one log store this way, each under its own names. G8 reads a name back only when it is exactly the environment's prefix followed by a real date. A day of dev-eu is never read as a day of dev. G8 also keeps two environments' days from being confused. A new slug is refused when its days would match another environment's names, or theirs would match its own, as dev-access would beside dev.

What G8 ships to the store​

The store has to be prepared once, as the installation guides show, with two things G8 ships.

  • The ingest pipeline g8-access sets each record's id to its request id. A record the gateway delivers twice is then stored once.
  • The index template g8-access applies to every index named g8-*-access-*. It says how each field is stored and searched, and sends every record through the pipeline.

The Turn on traffic logging dialog refuses to turn logging on while the store lacks either one or holds another version of it, and names the one it lacks. The Settings tab of Traffic Explorer shows whether the store holds both.

Reading the records​

The console never asks the log store itself. G8's server does, with the address and account a platform administrator entered in the environment's settings. It names each day of the window it reads, by its index name, with a day more on each side. It never searches a pattern that could reach another environment's days.

Deleting old days​

Each environment keeps its records for the number of days its retention says. Once a day, G8's server lists the environment's days in the store and picks those older than the retention. It deletes them whole, each by its own index name, never by a pattern.

The deletion is recorded the way a change in the console is. Before the first day is deleted, an audit row is written as pending, naming the days it is about to delete. When that row cannot be written, nothing is deleted. When the run ends, the row is completed with the days actually deleted and their size, or as failed with the reason. A run with nothing to delete leaves no row. A run that cannot list the days leaves a failed row. How the console writes to the gateway explains the pending row.

Loggers you add yourself​

A logger you add to a route or a service sends a second copy of the record somewhere else: elasticsearch-logger to an index of your own, gateway-access for example, or kafka-logger to a Kafka topic. G8's own records still go to the environment's days through g8-request-records, and Traffic Explorer reads only those.

Two limits hold.

  • An elasticsearch-logger whose index could name the days of another environment is refused when you save it. The index is read the way the gateway builds it for each request, with its time format and its variables.
  • A format of your own on G8's rule changes the records G8 reads. On that rule the form warns in these words: On the global rule g8 adds for its own request records, a format of your own replaces the record g8 reads and Traffic Explorer cannot read this environment's records; a copy on a route or service may use any format.
G8 dev