syslog

Sends connection logs to a syslog server.

Category ObservabilityAdd it to Stream routeEngine plugin syslogPriority 401

What it does

syslog sends a record of each connection a stream route carries to a syslog server at Host and Port, over tcp or udp as Socket type says. A connection has no default record. The record holds the fields of a Log format, set here or in the plugin metadata, and without one nothing is sent.

The lines are held back on the gateway node until Flush limit bytes have built up, 4096 unless you change it. They always travel in plain text, whatever TLS says.

When to use it

• The operations team gathers every log on one syslog server, and the connections MQTT clients open to the brokers on 1883 should join them there.
• A security team keeps a trail of which client addresses opened a database port, and how many bytes went each way.
• Every stream route should send the same fields, the client, the bytes received and sent and the time, set once in the plugin metadata.

Not for records of the HTTP requests a route takes — see syslog (Sends request logs to a syslog server).

Set it up

Send a record of each connection to a syslog server over TCP

A flush limit of 1 sends each record as soon as it is written instead of holding lines back until 4 KB have built up.

Set Host to syslog.internal, set Port to 514, leave Socket type at tcp, set Pool size to 10, fill Log format as in the JSON below and set Flush limit to 1.

syslog · as the console saves it

{
  "host": "syslog.internal",
  "port": 514,
  "sock_type": "tcp",
  "flush_limit": 1,
  "pool_size": 10,
  "log_format": {
    "client": "$remote_addr",
    "sent": "$bytes_sent",
    "at": "$time_iso8601"
  }
}

More scenarios

Shape the record every stream route sends to syslogenvironment-wide settings

Fill Log format as in the JSON below.

syslog environment-wide settings · as the console saves it

{
  "log_format": {
    "client": "$remote_addr",
    "received": "$bytes_received",
    "sent": "$bytes_sent",
    "at": "$time_iso8601"
  }
}

Things to know

• TLS — Turning this on changes nothing: the socket is opened under a name this value never reaches, so the lines always travel in plain text.
• Log format — A stream connection has no default record: unless a log format is set here, or in the syslog metadata that HTTP and stream share and the syslog plugin's page edits, nothing is sent.
• Log format — With sent taken from $bytes_sent, the record of an MQTT connection whose broker answered with 4 bytes carries 4 under sent.
• A log format in the plugin metadata reaches the records of a stream route whose syslog names none of its own. An MQTT client that sent 14 bytes and received 4 is recorded with 14 under received and 4 under sent.
• Flush limit — With Flush limit at 1, each connection's record reaches the server over TCP once it is written, without waiting for later connections.
• Flush limit — The drop limit has to stand above the flush limit: at or under it the socket logger never starts and no line is ever sent.
• Port — A port written as text is refused when you save.
• Socket type — Only tcp and udp are accepted; tls is refused when you save.
• Pool size — A pool below five connections, 4 for example, is refused when you save.
• AdviceOn a stream route with few connections, set a low Flush limit, as the example does, and keep Drop limit well above it.
• Drop limit — The drop limit has to stand above the flush limit: at or under it the socket logger never starts and no line is ever sent.

Settings

The fields as the console shows them, section by section. A field under “When … is …” is there only while that value is set.

HosthoststringRequired

The syslog server's host name or address.

PortportintegerRequired

The port the syslog server listens on.

Socket typesock_typestringdefault tcp

Whether entries travel over TCP or UDP.

one of tcp, udp

When Socket type is tcp

Pool sizepool_sizeintegerdefault 5

How many connections to the server are kept open for reuse.

at least 5

TLStlsbooleandefault false

Log formatlog_formatobject

The fields written in each entry, each a name and the connection variable it takes its value from. Without one here or in the shared metadata, nothing is sent.

example {"client":"$remote_addr","sent":"$bytes_sent","at":"$time_iso8601"}

Namenamestringdefault stream sys logger

A name for this logger, so its connections are kept apart from other loggers' in the pool.

Timeouttimeoutintegerdefault 3000

How long a send may take before it is given up.

at least 1 · in milliseconds

Batch sizebatch_max_sizeintegerdefault 1000

How many entries are sent together in one batch.

at least 1

Buffer durationbuffer_durationintegerdefault 60

How long entries may wait for a batch to fill before it is sent anyway.

at least 1 · in seconds

Inactive timeoutinactive_timeoutintegerdefault 5

How long the buffer may sit idle before it is flushed.

at least 1 · in seconds

Retriesmax_retry_countintegerdefault 0

How many times a failed send is retried.

at least 0

Retry delayretry_delayintegerdefault 1

How long to wait between retries.

at least 0 · in seconds

Flush limitflush_limitintegerdefault 4096

The buffered size, in bytes, at which entries are flushed to the server.

at least 1

Drop limitdrop_limitintegerdefault 1048576

The buffered size, in bytes, at which further entries are dropped.

Environment-wide settings

Set once per environment, under Plugins › syslog. Every syslog in the environment uses these.

Log formatlog_formatobject

Pending entry limitmax_pending_entriesintegerdefault 8192

How many log entries may wait to be sent. While the backlog is above this number, new entries are dropped rather than queued.

at least 1

Every plugin also has Enabled, Priority, Filter, Error response and Pre-function: see Plugin execution settings.