splunk-hec-logging

Sends request logs to a Splunk HTTP Event Collector.

Category ObservabilityAdd it to Route, Service, Consumer, Consumer group, Global rule, Plugin configEngine plugin splunk-hec-loggingPriority 409

What it does

splunk-hec-logging sends the records of the requests that reach the route or other object it is added to, to a Splunk HTTP Event Collector. The records are collected on the gateway node and sent in batches, each one a POST to the URI carrying the Token, so one call can carry several events.

The URI is the whole collector address, its path included, for example https://splunk.internal:8088/services/collector. Without a Log format here or in the plugin metadata, each event carries a fixed summary of the request and the response.

When to use it

• The security team searches the logs of the whole company in Splunk, and the request records of the orders API should be searchable there too.
• The operations team has created an HTTP Event Collector token for the gateway, and the gateway's records should arrive under it.
• Every route should send the same few fields, the client, method, path, status and time, set once in the plugin metadata.
• The collector expects the events on a channel of their own, named with each batch.

Not for records sent to an HTTP collector of your own — see http-logger (Sends request logs to an HTTP endpoint in batches), records kept in Elasticsearch — see elasticsearch-logger (Sends request logs to an Elasticsearch index) or records sent to a syslog server — see syslog (Sends request logs to a syslog server).

Set it up

Send request records to a Splunk HTTP Event Collector

Under Basics fill Endpoint as in the JSON below.

splunk-hec-logging · as the console saves it

{
  "endpoint": {
    "uri": "https://splunk.internal:8088/services/collector",
    "token": "<secret>"
  }
}

More scenarios

Shape the event every route sends to Splunkenvironment-wide settings

Fill Log format as in the JSON below.

splunk-hec-logging environment-wide settings · as the console saves it

{
  "log_format": {
    "client": "$remote_addr",
    "method": "$request_method",
    "path": "$uri",
    "status": "$status",
    "at": "$time_iso8601"
  }
}

Things to know

• URI — The events arrive as a POST to the path of this address, /services/collector in the example.
• URI — An address without its scheme, http:// or https://, is refused when you save.
• URI — The whole collector address, its path included: nothing is added to it.
• Token — A token is required: an endpoint without one is refused when you save.
• Token — An empty token is accepted, and the batch then travels with an empty credential.
• Keepalive timeout — At least 1000 milliseconds: 999 is refused when you save.
• A Log format in the plugin metadata shapes the event of every route whose splunk-hec-logging names no Log format of its own. In the example each event carries path with the path the client asked for.
• Log format — With no format here or in the plugin's metadata the event carries a fixed request and response summary, and the extra below adds only the keys that summary does not carry.
• Log format extra — Added to the default record only: while a log format is set, on this object or in the plugin's metadata, the engine writes that format alone and reads nothing here.
• Batch size — A batch goes out when it holds Batch size records, when its first record has waited Buffer duration, or when nothing new arrived for Inactive timeout.
• In the plugin metadata, Pending entry limit caps the backlog. While more entries than this wait to be sent, new ones are dropped rather than queued.
• AdviceGive the collector an https:// address, as the example does, and keep SSL verify on.

Settings

The fields as the console shows them, section by section.

Basics

EndpointendpointobjectRequired

Tokenendpoint.tokenstringRequired

The HEC token, sent as Splunk followed by the token with every batch. An empty token is accepted, and the batch then travels with an empty credential. Stored on the engine and shown only to roles that manage this object.

URIendpoint.uristringRequired

pattern

^[^\/]+:\/\/([\da-zA-Z.-]+|\[[\da-fA-F:]+\])(:\d+)?

· example https://splunk.internal:8088/services/collector

Channelendpoint.channelstring

Sent as the X-Splunk-Request-Channel header, and when empty no such header is sent.

Keepalive timeoutendpoint.keepalive_timeoutintegerdefault 60000

at least 1000 · in milliseconds

Timeoutendpoint.timeoutintegerdefault 10

at least 1 · in seconds

Behavior

Log formatlog_formatobject

example {"host":"$host","client":"$remote_addr","status":"$status","at":"$time_iso8601"}

Log format extralog_format_extraobject

example {"zone":"eu-west","gateway":"$hostname"}

SSL verifyssl_verifybooleandefault true

Advanced

Batch sizebatch_max_sizeintegerdefault 1000

at least 1

Buffer durationbuffer_durationintegerdefault 60

at least 1 · in seconds

Inactive timeoutinactive_timeoutintegerdefault 5

at least 1 · in seconds

Retriesmax_retry_countintegerdefault 0

at least 0

Retry delayretry_delayintegerdefault 1

at least 0 · in seconds

Namenamestringdefault splunk-hec-logging

Environment-wide settings

Set once per environment, under Plugins › splunk-hec-logging. Every splunk-hec-logging in the environment uses these.

Log formatlog_formatobject

example {"host":"$host","client":"$remote_addr","status":"$status","at":"$time_iso8601"}

Log format extralog_format_extraobject

example {"zone":"eu-west","gateway":"$hostname"}

Pending entry limitmax_pending_entriesintegerdefault 8192

How many log entries may wait to be sent. While the backlog is above this number, new entries are dropped rather than queued.

at least 1

Every plugin also has Enabled, Priority, Filter, Error response and Pre-function: see Plugin execution settings.