clickhouse-logger

Sends request logs to a ClickHouse table.

Category ObservabilityAdd it to Route, Service, Consumer, Consumer group, Global rule, Plugin configEngine plugin clickhouse-loggerPriority 398

What it does

clickhouse-logger stores the records of the requests that reach the route or other object it is added to in a ClickHouse table, through the server's HTTP interface. The records are collected on the gateway node and sent in batches, each one an INSERT into the Logtable, so one statement can carry several records.

Under One address every batch goes to the Endpoint address, under Several addresses to one address of the list picked at random. Every address ends in its path, / for the server's root, or nothing is stored.

When to use it

• The analytics team keeps request records in ClickHouse and queries them with SQL beside the rest of the company's data.
• The orders API should write one row per request into the table access_log of the database gateway.
• Several ClickHouse servers take records, and each batch may go to any one of them.
• Every route that stores its records in ClickHouse should write the same columns, the host, client, method, path, status and time, set once in the plugin metadata.

Not for records sent to an HTTP collector of your own — see http-logger (Sends request logs to an HTTP endpoint in batches), records kept in Elasticsearch — see elasticsearch-logger (Sends request logs to an Elasticsearch index) or records sent to a Kafka topic — see kafka-logger (Sends request logs to a Kafka topic).

Set it up

Store a record of each request in a ClickHouse table

The address ends in its path, / for the server's root; without one the gateway sends no request target and stores nothing. The password travels with every batch, so the address is the server's HTTPS port.

Under Basics choose One address, set Endpoint address to https://clickhouse.example.com:8443/, set Database to gateway, set Logtable to access_log, set User to gateway and set Password to <secret>.

clickhouse-logger · as the console saves it

{
  "endpoint_addr": "https://clickhouse.example.com:8443/",
  "database": "gateway",
  "logtable": "access_log",
  "user": "gateway",
  "password": "<secret>"
}

More scenarios

Spread the records over several ClickHouse servers

Each address ends in its path, / for the server's root; without one the gateway sends no request target and stores nothing. The password travels with every batch, so the addresses are the servers' HTTPS port.

Under Basics choose Several addresses, fill Endpoint addresses as in the JSON below, set Database to gateway, set Logtable to access_log, set User to gateway and set Password to <secret>.

clickhouse-logger · as the console saves it

{
  "endpoint_addrs": [
    "https://10.0.4.21:8443/",
    "https://10.0.4.22:8443/"
  ],
  "database": "gateway",
  "logtable": "access_log",
  "user": "gateway",
  "password": "<secret>"
}

Choose the fields every route that carries the plugin writes into its recordenvironment-wide settings

Fill Log format as in the JSON below.

clickhouse-logger environment-wide settings · as the console saves it

{
  "log_format": {
    "host": "$host",
    "client": "$remote_addr",
    "method": "$request_method",
    "uri": "$uri",
    "status": "$status",
    "at": "$time_iso8601"
  }
}

Things to know

• Endpoint address — An address that stops at the port sends a request with no target, and nothing is stored. End it with / for the server's root, as the example does.
• Endpoint address — An address without its scheme, 127.0.0.1:8123 for example, is refused when you save.
• Endpoint addresses — Give either one address or a list, never both and never neither; either way the save is refused.
• Endpoint addresses — Every address must accept the same records: each batch goes to one of them at random.
• Logtable — Each batch arrives at the server as INSERT INTO access_log FORMAT JSONEachRow, where access_log is the table named here.
• Logtable — The name is pasted into the INSERT the batch is sent as, so an empty one sends a statement with no table at all.
• A Log format in the plugin metadata gives the same record to every route whose clickhouse-logger names no Log format of its own. In the example each record carries at with the time of the request.
• Log format extra — Added to the default record only: while a log format is set, on this object or in the plugin's metadata, the engine writes that format alone and reads nothing here.
• Timeout — A whole number of seconds, at least one: 0 is refused when you save.
• Include request body when — Not checked at save: an expression the engine cannot read is stored, and the body is then never logged. Only the gateway's error log says so.
• Include response body when — Not checked at save: an expression the engine cannot read is stored, and the body is then never logged. Only the gateway's error log says so.
• Batch size — A batch goes out when it holds Batch size records, when its first record has waited Buffer duration, or when nothing new arrived for Inactive timeout.
• In the plugin metadata, Pending entry limit caps the backlog. While more entries than this wait to be sent, new ones are dropped rather than queued.
• AdviceGive an https:// address whenever a Password is set, as the examples do.
• AdviceTurn on the request and response bodies only on routes whose table may hold what clients and the backend send.

Settings

The fields as the console shows them, section by section.

Basics

Choose one:

• One addressEvery batch goes to this address.
• Several addressesEvery batch goes to one address picked from the list at random.

One address

Endpoint addressendpoint_addrstringRequired

The address of the ClickHouse HTTP interface, ending with a path such as http://clickhouse.example.com:8123/. An address that stops at the port sends a request with no target, and nothing is stored.

pattern

^[^\/]+:\/\/([\da-zA-Z.-]+|\[[\da-fA-F:]+\])(:\d+)?

· example https://clickhouse.example.com:8443/

Several addresses

Endpoint addressesendpoint_addrsarray of stringsRequired

The addresses of the ClickHouse HTTP interfaces, each ending with a path such as http://clickhouse.example.com:8123/. An address that stops at the port sends a request with no target, and nothing is stored.

at least 1 item

Databasedatabasestringdefault

Logtablelogtablestringdefault

Useruserstringdefault

Passwordpasswordstringdefault

Stored on the engine and shown only to roles that manage this object.

Behavior

Timeouttimeoutintegerdefault 3

at least 1 · in seconds

SSL verifyssl_verifybooleandefault true

Log formatlog_formatobject

example {"host":"$host","client":"$remote_addr","status":"$status","at":"$time_iso8601"}

Log format extralog_format_extraobject

example {"zone":"eu-west","gateway":"$hostname"}

Include request bodyinclude_req_bodybooleandefault false

Include request body wheninclude_req_body_exprarray

at least 1 item · conditions on request variables: see Conditions on request variables

Include response bodyinclude_resp_bodybooleandefault false

Include response body wheninclude_resp_body_exprarray

at least 1 item · conditions on request variables: see Conditions on request variables

Request body limitmax_req_body_bytesintegerdefault 524288

at least 1

Response body limitmax_resp_body_bytesintegerdefault 524288

at least 1

Advanced

Batch sizebatch_max_sizeintegerdefault 1000

at least 1

Buffer durationbuffer_durationintegerdefault 60

at least 1 · in seconds

Inactive timeoutinactive_timeoutintegerdefault 5

at least 1 · in seconds

Retriesmax_retry_countintegerdefault 0

at least 0

Retry delayretry_delayintegerdefault 1

at least 0 · in seconds

Namenamestringdefault clickhouse-logger

Environment-wide settings

Set once per environment, under Plugins › clickhouse-logger. Every clickhouse-logger in the environment uses these.

Log formatlog_formatobject

example {"host":"$host","client":"$remote_addr","status":"$status","at":"$time_iso8601"}

Log format extralog_format_extraobject

example {"zone":"eu-west","gateway":"$hostname"}

Pending entry limitmax_pending_entriesintegerdefault 8192

How many log entries may wait to be sent. While the backlog is above this number, new entries are dropped rather than queued.

at least 1

Every plugin also has Enabled, Priority, Filter, Error response and Pre-function: see Plugin execution settings.