elasticsearch-logger

Sends request logs to an Elasticsearch index.

Category ObservabilityAdd it to Route, Service, Consumer, Consumer group, Global rule, Plugin configEngine plugin elasticsearch-loggerPriority 413

What it does

elasticsearch-logger sends the records of the requests that reach the route or other object it is added to, to the Elasticsearch index named under Index. The records are collected on the gateway node and sent in batches, each one a POST to the /_bulk path of the address.

When traffic logging is on, g8 adds a copy of its own to a global rule, and Traffic Explorer reads the records it writes. A copy you add to a route or service is separate from that one and may use any Log format.

When to use it

• The operations team searches the records of one route, the orders API for example, in an Elasticsearch of its own.
• Each day's records should land in an index of their own, so a whole day can be searched or removed at once.
• The cluster has three nodes and any of them accepts records, so a batch may go to whichever of them.
• The cluster asks for a user and password with the right to write the index.

Not for records sent to an HTTP endpoint — see http-logger (Sends request logs to an HTTP endpoint in batches), records sent to a Kafka topic — see kafka-logger (Sends request logs to a Kafka topic) or records written to a file on the gateway node — see file-logger (Writes request logs to a file on the engine's node).

Guides

Pages that walk through a job with elasticsearch-logger, step by step.

• Send request records to ElasticsearchSend a record of every request a route takes to an Elasticsearch index of your own.

Set it up

Index a record of each request in Elasticsearch

A copy added here is separate from the one g8 adds for its own request records when traffic logging is on.

Under Basics choose One address, set Endpoint address to https://elasticsearch.example.com:9200, fill Field as in the JSON below and fill Auth as in the JSON below.

The console's Add plugin panel, filled with this example.

elasticsearch-logger · as the console saves it

{
  "endpoint_addr": "https://elasticsearch.example.com:9200",
  "field": {
    "index": "gateway-access"
  },
  "auth": {
    "username": "gateway",
    "password": "<secret>"
  }
}

More scenarios

Send the records to any node of an Elasticsearch cluster

A copy added here is separate from the one g8 adds for its own request records when traffic logging is on.

Under Basics choose Several addresses, fill Endpoint addresses as in the JSON below, fill Field as in the JSON below and fill Auth as in the JSON below.

elasticsearch-logger · as the console saves it

{
  "endpoint_addrs": [
    "https://10.0.6.31:9200",
    "https://10.0.6.32:9200",
    "https://10.0.6.33:9200"
  ],
  "field": {
    "index": "gateway-access-{%Y.%m.%d}"
  },
  "auth": {
    "username": "gateway",
    "password": "<secret>"
  }
}

Drop new records once 50000 are waiting to be sent from any routeenvironment-wide settings

The cap is read by every use of the plugin, the copy g8 adds for its own request records included, so it replaces the default of 8192 there too.

Set Pending entry limit to 50000.

elasticsearch-logger environment-wide settings · as the console saves it

{
  "max_pending_entries": 50000
}

Things to know

• Endpoint address — The records arrive as a POST to /_bulk under this address.
• Endpoint address — The bulk path is added to this address, which is why one that ends in a slash is refused.
• Endpoint addresses — The bulk path is added to this address, which is why one that ends in a slash is refused.
• Index — With gateway-access-{%Y.%m.%d}, each record is written to the index of its day, gateway-access- followed by the year, the month and the day.
• Index — The name is built again for every request: braces hold a time format and a $name is a request variable.
• Auth — Username and Password go together: one without the other is refused when you save.
• Headers — These are written over the headers the plugin sends: an Authorization here replaces the one built from the credential above, and a content type here replaces the bulk one.
• Timeout — Before logging, the address is asked for its version. While it does not answer, each request waits this long and the next one asks again.
• Log format — On the global rule g8 adds for its own request records, a format of your own replaces the record g8 reads and Traffic Explorer cannot read this environment's records; a copy on a route or service may use any format.
• Log format — A format of your own replaces the record g8 reads: while traffic logging is on, the request records of this environment cannot be read in Traffic Explorer.
• Log format extra — Added to the default record only: while a log format is set, on this object or in the plugin's metadata, the engine writes that format alone and reads nothing here.
• Batch size — A batch goes out when it holds Batch size records, when its first record has waited Buffer duration, or when nothing new arrived for Inactive timeout.
• Pending entry limit — While more entries than this wait to be sent, new ones are dropped rather than queued.
• Include request body when — Not checked at save: an expression the engine cannot read is stored, and the body is then never logged. Only the gateway's error log says so.
• Include response body when — Not checked at save: an expression the engine cannot read is stored, and the body is then never logged. Only the gateway's error log says so.
• AdviceGive the account under Auth the right to write the indexes this copy names and nothing more.

Settings

The fields as the console shows them, section by section.

Basics

Choose one:

• One addressEvery batch goes to this address.
• Several addressesEvery batch goes to one address picked from the list at random.

One address

Endpoint addressendpoint_addrstringRequired

pattern

[^/]$

· example https://elasticsearch.internal:9200

Several addresses

Endpoint addressesendpoint_addrsarray of stringsRequired

Every address must accept the same records: each batch goes to one of them at random.

at least 1 item

FieldfieldobjectRequired

Indexfield.indexstringRequired

Authauthobject

Passwordauth.passwordstringRequired

Stored on the engine and shown only to roles that manage this object.

at least 1 character

Usernameauth.usernamestringRequired

at least 1 character

Behavior

Headersheadersobject

Stored on the engine and shown only to roles that manage this object. An object whose keys are a name without a colon, each with a string as its value.

at least 1 key

Timeouttimeoutintegerdefault 10

at least 1 · in seconds

SSL verifyssl_verifybooleandefault true

Log formatlog_formatobject

example {"host":"$host","client":"$remote_addr","status":"$status","at":"$time_iso8601"}

Log format extralog_format_extraobject

example {"zone":"eu-west","gateway":"$hostname"}

Include request bodyinclude_req_bodybooleandefault false

Include request body wheninclude_req_body_exprarray

at least 1 item · conditions on request variables: see Conditions on request variables

Include response bodyinclude_resp_bodybooleandefault false

Include response body wheninclude_resp_body_exprarray

at least 1 item · conditions on request variables: see Conditions on request variables

Request body limitmax_req_body_bytesintegerdefault 524288

at least 1

Response body limitmax_resp_body_bytesintegerdefault 524288

at least 1

Advanced

Batch sizebatch_max_sizeintegerdefault 1000

at least 1

Buffer durationbuffer_durationintegerdefault 60

at least 1 · in seconds

Inactive timeoutinactive_timeoutintegerdefault 5

at least 1 · in seconds

Retriesmax_retry_countintegerdefault 0

at least 0

Retry delayretry_delayintegerdefault 1

at least 0 · in seconds

Namenamestringdefault elasticsearch-logger

Environment-wide settings

Set once per environment, under Plugins › elasticsearch-logger. Every elasticsearch-logger in the environment uses these.

Log formatlog_formatobject

example {"host":"$host","client":"$remote_addr","status":"$status","at":"$time_iso8601"}

Log format extralog_format_extraobject

example {"zone":"eu-west","gateway":"$hostname"}

Pending entry limitmax_pending_entriesintegerdefault 8192

How many log entries may wait to be sent. While the backlog is above this number, new entries are dropped rather than queued.

at least 1

Every plugin also has Enabled, Priority, Filter, Error response and Pre-function: see Plugin execution settings.