loki-logger

Sends request logs to Grafana Loki.

Category ObservabilityAdd it to Route, Service, Consumer, Consumer group, Global rule, Plugin configEngine plugin loki-loggerPriority 414

What it does

loki-logger sends the records of the requests that reach the route or other object it is added to, to Grafana Loki at one of the Endpoint addresses. The records are collected on the gateway node and sent in batches, each one a POST to /loki/api/v1/push on that address unless Endpoint URI names another path.

Each batch carries the Tenant ID in its own header and the Log labels as the labels of its stream. A label value written as a variable, $route_id for example, is read on each request, so the records of each route can be told apart in Loki.

When to use it

• The operations team reads the logs of every service in Grafana Loki, and the request records of the orders API should join them there.
• Several teams share one Loki, each under a tenant of its own, and the gateway's records belong under the tenant gateway.
• Records should be found in Loki by route: a label set to $route_id gives each route a label value of its own.
• Every record in the environment should carry the name of the environment and the gateway host, set once in the plugin metadata.

Not for records sent to an HTTP collector of your own — see http-logger (Sends request logs to an HTTP endpoint in batches), records kept in Elasticsearch — see elasticsearch-logger (Sends request logs to an Elasticsearch index) or records sent to a Kafka topic — see kafka-logger (Sends request logs to a Kafka topic).

Set it up

Push request records to Loki under one tenant with a label for the route

Under Basics fill Endpoint addresses as in the JSON below, set Tenant ID to gateway and fill Log labels as in the JSON below.

loki-logger · as the console saves it

{
  "endpoint_addrs": [
    "http://loki.internal:3100"
  ],
  "tenant_id": "gateway",
  "log_labels": {
    "job": "gateway",
    "route": "$route_id"
  }
}

More scenarios

Add the environment and the gateway host to every recordenvironment-wide settings

Fill Log format extra as in the JSON below.

loki-logger environment-wide settings · as the console saves it

{
  "log_format_extra": {
    "environment": "production",
    "gateway": "$hostname"
  }
}

Things to know

• Endpoint addresses — An address without its scheme, 127.0.0.1:3100 for example, is refused when you save.
• Endpoint addresses — Every address must accept the same records: each batch goes to one of them at random.
• Tenant ID — The tenant arrives at Loki as the X-Scope-OrgID header of each push, gateway in the example.
• Tenant ID — This travels as the X-Scope-OrgID header and is written over a header of that name below.
• Log labels — The labels travel with the pushed records: the example's job label arrives as gateway.
• Log labels — A label value is text: a number, 7 for example, is refused when you save.
• Log labels — Values are read per request, so $uri becomes each request's path. Entries with equal labels travel as one stream.
• Endpoint URI — Left as the form starts it, the records arrive as a POST to /loki/api/v1/push on the address.
• Endpoint URI — Start it with a slash. Without one the batch goes out with a malformed path and the server does not find its push endpoint.
• Headers — Content-Type is always sent as application/json and X-Scope-OrgID as the tenant above. Other headers travel as written.
• A Log format extra in the plugin metadata adds its fields to the records of every route whose loki-logger names no Log format of its own. In the example each record carries environment set to production, and gateway with the gateway's host name.
• Log format extra — Added to the default record only: while a log format is set, on this object or in the plugin's metadata, the engine writes that format alone and reads nothing here.
• Timeout — At most 60000 milliseconds: 60001 is refused when you save.
• Include request body when — The console’s form draws one level of conditions; a value that nests deeper is kept as it is and edited in Raw JSON.
• Include response body when — The console’s form draws one level of conditions; a value that nests deeper is kept as it is and edited in Raw JSON.
• Batch size — A batch goes out when it holds Batch size records, when its first record has waited Buffer duration, or when nothing new arrived for Inactive timeout.
• Keepalive timeout — At least 1000 milliseconds: 999 is refused when you save.
• In the plugin metadata, Pending entry limit caps the backlog. While more entries than this wait to be sent, new ones are dropped rather than queued.
• AdviceTurn on the request and response bodies only on routes whose records may hold what clients and the backend send.

Settings

The fields as the console shows them, section by section.

Basics

Endpoint addressesendpoint_addrsarray of stringsRequired

at least 1 item

Tenant IDtenant_idstringdefault fake

Log labelslog_labelsobjectdefault {"job":"apisix"}

An object whose keys are any name, each with a string as its value.

Behavior

Endpoint URIendpoint_uristringdefault /loki/api/v1/push

at least 1 character

Headersheadersobject

Stored on the engine and shown only to roles that manage this object. An object whose keys are any name, each with a string as its value.

Timeouttimeoutintegerdefault 3000

1 to 60000 · in milliseconds

SSL verifyssl_verifybooleandefault false

Log formatlog_formatobject

example {"host":"$host","client":"$remote_addr","status":"$status","at":"$time_iso8601"}

Log format extralog_format_extraobject

example {"zone":"eu-west","gateway":"$hostname"}

Include request bodyinclude_req_bodybooleandefault false

Include request body wheninclude_req_body_exprarray

at least 1 item · conditions on request variables: see Conditions on request variables

Include response bodyinclude_resp_bodybooleandefault false

Include response body wheninclude_resp_body_exprarray

at least 1 item · conditions on request variables: see Conditions on request variables

Request body limitmax_req_body_bytesintegerdefault 524288

at least 1

Response body limitmax_resp_body_bytesintegerdefault 524288

at least 1

Advanced

Batch sizebatch_max_sizeintegerdefault 1000

at least 1

Buffer durationbuffer_durationintegerdefault 60

at least 1 · in seconds

Inactive timeoutinactive_timeoutintegerdefault 5

at least 1 · in seconds

Retriesmax_retry_countintegerdefault 0

at least 0

Retry delayretry_delayintegerdefault 1

at least 0 · in seconds

Keepalivekeepalivebooleandefault true

Keepalive poolkeepalive_poolintegerdefault 5

at least 1

Keepalive timeoutkeepalive_timeoutintegerdefault 60000

at least 1000 · in milliseconds

Namenamestringdefault loki logger

Environment-wide settings

Set once per environment, under Plugins › loki-logger. Every loki-logger in the environment uses these.

Log formatlog_formatobject

example {"host":"$host","client":"$remote_addr","status":"$status","at":"$time_iso8601"}

Log format extralog_format_extraobject

example {"zone":"eu-west","gateway":"$hostname"}

Pending entry limitmax_pending_entriesintegerdefault 8192

How many log entries may wait to be sent. While the backlog is above this number, new entries are dropped rather than queued.

at least 1

Every plugin also has Enabled, Priority, Filter, Error response and Pre-function: see Plugin execution settings.