http-logger

Sends request logs to an HTTP endpoint in batches.

Category ObservabilityAdd it to Route, Service, Consumer, Consumer group, Global rule, Plugin configEngine plugin http-loggerPriority 410

What it does

http-logger sends the records of the requests that reach the route or other object it is added to, to an HTTP endpoint, the URI. The records are collected on the gateway node and sent in batches, each one a POST to that address, so one call to the endpoint can carry several records.

A Log format of your own replaces the default record with the fields you name. It is set here or, for the whole environment, in the plugin metadata. Log format extra adds fields to the default record only, and is not read while a Log format is set.

When to use it

• Request records have to reach a collector the operations team runs behind an HTTP endpoint, for example https://logs.example.com/gateway/entries.
• A small service of your own should receive the records of one route, the orders API for example, to keep its own audit of who called it.
• Every record in the environment should carry the same few fields, the client address, method, path, status and backend time, set once in the plugin metadata.
• The collector accepts records only with an Authorization header that carries a token.

Not for records sent to a Kafka topic — see kafka-logger (Sends request logs to a Kafka topic), records kept in Elasticsearch — see elasticsearch-logger (Sends request logs to an Elasticsearch index) or records sent to a syslog server — see syslog (Sends request logs to a syslog server).

Set it up

Send request records in batches to an HTTP collector

auth_header is the whole Authorization value the collector receives, scheme included, so for a bearer token write Bearer, a space and the token.

Under Basics set URI to https://logs.example.com/gateway/entries and set Auth header to <secret>. Under Behavior set Timeout to 5 seconds and turn on SSL verify. Under Advanced set Batch size to 500 and set Buffer duration to 30 seconds.

http-logger · as the console saves it

{
  "uri": "https://logs.example.com/gateway/entries",
  "auth_header": "<secret>",
  "batch_max_size": 500,
  "buffer_duration": 30,
  "timeout": 5,
  "ssl_verify": true
}

More scenarios

Choose the fields every request record carriesenvironment-wide settings

Fill Log format as in the JSON below.

http-logger environment-wide settings · as the console saves it

{
  "log_format": {
    "client": "$remote_addr",
    "method": "$request_method",
    "path": "$uri",
    "status": "$status",
    "upstream_seconds": "$upstream_response_time"
  }
}

Things to know

• URI — The records arrive as a POST to the path of this address, /gateway/entries in the example.
• URI — An address without its scheme, http:// or https://, is refused when you save.
• Auth header — The value is the whole Authorization header the collector receives, scheme included. For a bearer token write Bearer, a space and the token.
• Batch size — A batch goes out when it holds Batch size records, when its first record has waited Buffer duration, or when nothing new arrived for Inactive timeout.
• Concat method — Only json and new_line are accepted; csv is refused when you save.
• Timeout — A whole number of seconds: a fraction, 1.5 for example, is refused when you save.
• A log format in the plugin metadata reaches the records of a route whose http-logger names none of its own. A GET is then recorded with method set to GET, and every record carries upstream_seconds.
• Log format extra — Added to the default record only: while a log format is set, on this object or in the plugin's metadata, the engine writes that format alone and reads nothing here.
• Pending entry limit — While more entries than this wait to be sent, new ones are dropped rather than queued.
• Include request body when — The console’s form draws one level of conditions; a value that nests deeper is kept as it is and edited in Raw JSON.
• Include response body when — The console’s form draws one level of conditions; a value that nests deeper is kept as it is and edited in Raw JSON.
• Auth header — Each batch reaches the collector with an Authorization header carrying this value.
• AdviceTurn on the request and response bodies only on routes whose collector is meant to hold what clients and the backend send.

Settings

The fields as the console shows them, section by section.

Basics

URIuristringRequired

pattern

^[^\/]+:\/\/([\da-zA-Z.-]+|\[[\da-fA-F:]+\])(:\d+)?

· example http://logs.internal:8080/entries

Auth headerauth_headerstring

Stored on the engine and shown only to roles that manage this object.

Behavior

Concat methodconcat_methodstringdefault json

one of json, new_line

Timeouttimeoutintegerdefault 3

at least 1 · in seconds

SSL verifyssl_verifybooleandefault false

Log formatlog_formatobject

example {"host":"$host","client":"$remote_addr","status":"$status","at":"$time_iso8601"}

Log format extralog_format_extraobject

example {"zone":"eu-west","gateway":"$hostname"}

Include request bodyinclude_req_bodybooleandefault false

Include request body wheninclude_req_body_exprarray

at least 1 item · conditions on request variables: see Conditions on request variables

Include response bodyinclude_resp_bodybooleandefault false

Include response body wheninclude_resp_body_exprarray

at least 1 item · conditions on request variables: see Conditions on request variables

Request body limitmax_req_body_bytesintegerdefault 524288

at least 1

Response body limitmax_resp_body_bytesintegerdefault 524288

at least 1

Advanced

Batch sizebatch_max_sizeintegerdefault 1000

at least 1

Buffer durationbuffer_durationintegerdefault 60

at least 1 · in seconds

Inactive timeoutinactive_timeoutintegerdefault 5

at least 1 · in seconds

Retriesmax_retry_countintegerdefault 0

at least 0

Retry delayretry_delayintegerdefault 1

at least 0 · in seconds

Namenamestringdefault http logger

Environment-wide settings

Set once per environment, under Plugins › http-logger. Every http-logger in the environment uses these.

Log formatlog_formatobject

example {"host":"$host","client":"$remote_addr","status":"$status","at":"$time_iso8601"}

Log format extralog_format_extraobject

example {"zone":"eu-west","gateway":"$hostname"}

Pending entry limitmax_pending_entriesintegerdefault 8192

How many log entries may wait to be sent. While the backlog is above this number, new entries are dropped rather than queued.

at least 1

Every plugin also has Enabled, Priority, Filter, Error response and Pre-function: see Plugin execution settings.