Dev environment — 1 APISIX + 1 etcd on Docker

The host names, addresses and the registry on this page are examples. Our lab run used its own values in their place; replace them with yours, and take the registry address and its credentials from your G8 contact.

One APISIX and one etcd on a single machine, via Docker Compose. The smallest working install: for development, for experimentation, and for validating the install chain end to end.

This is not a highly available install

One etcd node, one APISIX. If the machine goes down, the environment goes down. For the production topology see the prod install.

1. What gets installed

component	version	source
APISIX	3.19.0-debian	`harbor.g8.dev/infra/apisix:3.19.0-debian`
etcd	v3.5.17	`harbor.g8.dev/infra/etcd:v3.5.17`

Images are pulled from our own Harbor registry. A customer site may have no internet access; we leave no dependency on upstream (Docker Hub, quay.io).

2. Prerequisites

requirement	value	note
Operating system	Ubuntu 24.04 LTS	The CA trust command differs on other distributions
CPU / RAM / disk	4 vCPU / 8 GB / 100 GB	etcd is fsync-sensitive; SSD/NVMe recommended
Time synchronisation	`chrony` or `systemd-timesyncd` active	etcd enters leader election on clock drift
Harbor access	`harbor.g8.dev:443`	Image source
DNS	`harbor.g8.dev` must resolve
Harbor CA file	`g8-lab-ca.crt` in the working directory	Shipped with the installation package; the CA step below copies this file
Internet access	`download.docker.com`, `archive.ubuntu.com`	Required. APISIX and etcd come from Harbor, but Docker and the OS packages are installed from upstream. A genuinely air-gapped site needs a local mirror for those steps

2.1 Ports

port	component	who should reach it
9080	APISIX data plane	Clients
9180	APISIX Admin API	The g8 server only. Never the internet
9091	APISIX Prometheus	Prometheus only
9090	APISIX control API	Container-local `127.0.0.1` only. The healthcheck in section 8 targets it; not published to the host
9200	Elasticsearch	The g8 server and the edge only. Authenticated, but there is no TLS — the password crosses the network in the clear
9090	Prometheus	The g8 server and the edge only. No authentication
2379	etcd client	APISIX only. Not published to the host
2380	etcd peer	Unused on a single node

Publishing the Admin API

The docker-compose.yml below publishes 9180 on every host interface; that is a lab convenience. In production bind it to the internal address only (127.0.0.1:9180:9180), or do not publish it at all and put the g8 server on the same Docker network.

3. Install Docker

The apt lock on cloud images

On a freshly provisioned Ubuntu cloud image, unattended-upgrades holds the apt locks during first boot and the apt-get install below fails silently — it returns a non-zero code, and if automation swallows that, nothing gets installed.

In automation, wait for all four locks before installing. Do not wait on the process name (pgrep unattended-upgrade): unattended-upgrade-shutdown --wait-for-signal runs permanently and will lock the loop forever.

for i in $(seq 1 90); do
  BUSY=0
  for L in /var/lib/dpkg/lock-frontend /var/lib/dpkg/lock \
           /var/lib/apt/lists/lock /var/cache/apt/archives/lock; do
    sudo fuser "$L" >/dev/null 2>&1 && BUSY=1
  done
  [ "$BUSY" = "0" ] && break
  sleep 10
done

sudo apt-get update -qq
sudo apt-get install -y ca-certificates curl gnupg jq
sudo install -m 0755 -d /etc/apt/keyrings
curl -fsSL https://download.docker.com/linux/ubuntu/gpg | sudo gpg --dearmor -o /etc/apt/keyrings/docker.gpg
sudo chmod a+r /etc/apt/keyrings/docker.gpg
echo "deb [arch=amd64 signed-by=/etc/apt/keyrings/docker.gpg] https://download.docker.com/linux/ubuntu noble stable" \
  | sudo tee /etc/apt/sources.list.d/docker.list >/dev/null
sudo apt-get update -qq
sudo apt-get install -y docker-ce docker-ce-cli containerd.io docker-buildx-plugin docker-compose-plugin
sudo systemctl enable --now docker

4. Trust the Harbor CA

Harbor presents a certificate signed by our own CA. Until that CA is trusted, no image can be pulled — you get x509: certificate signed by unknown authority.

sudo cp g8-lab-ca.crt /usr/local/share/ca-certificates/g8-lab-ca.crt
sudo update-ca-certificates
sudo mkdir -p /etc/docker/certs.d/harbor.g8.dev
sudo cp g8-lab-ca.crt /etc/docker/certs.d/harbor.g8.dev/ca.crt

Verify — you must see sslverify=0:

curl -s -o /dev/null -w "sslverify=%{ssl_verify_result} status=%{http_code}\n" \
  https://harbor.g8.dev/api/v2.0/systeminfo

sslverify=0 status=200

5. Directories

The APISIX image runs as uid 636. If the bind-mounted log directory belongs to another user, APISIX cannot start — details: the uid 636 trap.

sudo mkdir -p /opt/g8-gw/apisix/logs /opt/g8-gw/etcd-data /opt/g8-gw/backups
sudo chown -R $USER:$USER /opt/g8-gw
sudo chown -R 636:636 /opt/g8-gw/apisix/logs

6. Images

sudo docker pull harbor.g8.dev/infra/etcd:v3.5.17
sudo docker pull harbor.g8.dev/infra/apisix:3.19.0-debian

7. Admin key

APISIX ships with a documented default admin key (edd1c9f0…) that everyone on the internet has. Left unchanged, the Admin API is open with full privileges.

The key lives in .env and reaches the container as an environment variable; it is never typed into config.yaml by hand.

printf 'G8_APISIX_ADMIN_KEY=%s\n' "$(openssl rand -hex 24)" | sudo tee /opt/g8-gw/.env >/dev/null
sudo chmod 600 /opt/g8-gw/.env

8. Configuration files

/opt/g8-gw/apisix/config.yaml
sudo tee /opt/g8-gw/apisix/config.yaml >/dev/null <<'EOF'
apisix:
  node_listen:
    - 9080
  enable_admin: true

deployment:
  role: traditional
  role_traditional:
    config_provider: etcd
  admin:
    admin_listen:
      ip: 0.0.0.0
      port: 9180
    admin_key:
      - name: admin
        key: "${{G8_APISIX_ADMIN_KEY}}"
        role: admin
    allow_admin:
      - 192.0.2.0/24
      - 127.0.0.0/8
      - 172.16.0.0/12
  etcd:
    host:
      - "http://etcd:2379"
    prefix: /apisix
    timeout: 30

nginx_config:
  error_log_level: warn
  http:
    access_log: /usr/local/apisix/logs/access.log
    access_log_format: '{"time":"$time_iso8601","remote_addr":"$remote_addr","status":"$status","method":"$request_method","uri":"$uri","rt":"$request_time","ua":"$http_user_agent"}'
    access_log_format_escape: json

plugin_attr:
  prometheus:
    export_addr:
      ip: 0.0.0.0
      port: 9091
    metrics:
      http_status:
        extra_labels:
          - route_name: $route_name
          - service_name: $service_name
      http_latency:
        extra_labels:
          - route_name: $route_name
          - service_name: $service_name
EOF

${{G8_APISIX_ADMIN_KEY}} is APISIX's environment-variable syntax; it is filled in at startup from .env. 172.16.0.0/12 covers the Docker bridge network — details: the Docker source IP trap.

/opt/g8-gw/docker-compose.yml
sudo tee /opt/g8-gw/docker-compose.yml >/dev/null <<'EOF'
x-logging: &log-rotation
  driver: json-file
  options:
    max-size: "10m"
    max-file: "3"

services:
  etcd:
    image: harbor.g8.dev/infra/etcd:v3.5.17
    container_name: g8-etcd
    restart: unless-stopped
    command:
      - etcd
      - --name=etcd-dev
      - --data-dir=/etcd-data
      - --listen-client-urls=http://0.0.0.0:2379
      - --advertise-client-urls=http://etcd:2379
      - --listen-peer-urls=http://0.0.0.0:2380
      - --initial-advertise-peer-urls=http://etcd:2380
      - --initial-cluster=etcd-dev=http://etcd:2380
      - --initial-cluster-state=new
      - --initial-cluster-token=g8-dev
      - --auto-compaction-mode=periodic
      - --auto-compaction-retention=1
      - --quota-backend-bytes=2147483648
      - --snapshot-count=10000
    volumes:
      - /opt/g8-gw/etcd-data:/etcd-data
    healthcheck:
      test: ["CMD", "etcdctl", "endpoint", "health"]
      interval: 10s
      timeout: 5s
      retries: 5
      start_period: 20s
    logging: *log-rotation
    networks: [g8gw]

  apisix:
    image: harbor.g8.dev/infra/apisix:3.19.0-debian
    container_name: g8-apisix
    restart: unless-stopped
    env_file:
      - .env
    depends_on:
      etcd:
        condition: service_healthy
    volumes:
      - /opt/g8-gw/apisix/config.yaml:/usr/local/apisix/conf/config.yaml:ro
      - /opt/g8-gw/apisix/logs:/usr/local/apisix/logs
    ports:
      - "9080:9080"
      - "9180:9180"
      - "9091:9091"
    ulimits:
      nofile:
        soft: 65536
        hard: 65536
    healthcheck:
      test: ["CMD", "bash", "-c", "echo > /dev/tcp/127.0.0.1/9090"]
      interval: 10s
      timeout: 5s
      retries: 6
      start_period: 30s
    logging: *log-rotation
    networks: [g8gw]

networks:
  g8gw:
    name: g8gw
EOF

The 192.0.2.0/24 entry in the allow_admin list in the config.yaml above is this lab's network. Replace it with the network the g8 server sits on for a customer installation. This is the one mistake the document can stay silent about: if you leave it, the verification steps will not catch it (they test from 127.0.0.1), but in a customer environment even a request carrying the correct key gets a 403. 127.0.0.0/8 and 172.16.0.0/12 must stay — the reason is in the source IP trap.

The healthcheck targets 9090: that is APISIX's control API port, reachable only from inside the container (2.1). Why it does not target 9080 is in 16.7.

The rationale for --auto-compaction-*, --quota-backend-bytes and the log rotation settings is in long-term operation. Why the healthcheck does not use curl: the HTTP client trap.

9. Start

cd /opt/g8-gw && sudo docker compose up -d
S=$(sudo docker compose ps --services | wc -l)
for i in $(seq 1 20); do
  [ "$(sudo docker compose ps --format '{{.Status}}' | grep -c '(healthy)')" = "$S" ] && break
  sleep 8
done
echo "healthy=$(sudo docker compose ps --format '{{.Status}}' | grep -c '(healthy)')/$S unhealthy=$(sudo docker compose ps --format '{{.Status}}' | grep -c unhealthy)"

healthy=4/4 unhealthy=0

up -d starts the containers, it does not wait for them to become healthy. APISIX's start_period is 30 seconds; an operator who runs ps immediately afterwards sees health: starting and stops where the document prints (healthy).

10. Verification

10.1 Health

sudo docker compose ps --format "{{.Name}} | {{.Status}}"

g8-apisix | Up 2 minutes (healthy)
g8-etcd | Up 5 minutes (healthy)

10.2 Admin API and key substitution

A three-way test. The middle line is the critical one: if the raw template string is rejected, the environment variable really was substituted.

KEY=$(sudo grep G8_APISIX_ADMIN_KEY /opt/g8-gw/.env | cut -d= -f2)
curl -s -o /dev/null -H "X-API-KEY: $KEY" -w "gercek anahtar : %{http_code}\n" http://127.0.0.1:9180/apisix/admin/routes
curl -s -o /dev/null -H 'X-API-KEY: ${{G8_APISIX_ADMIN_KEY}}' -w "ham sablon     : %{http_code}\n" http://127.0.0.1:9180/apisix/admin/routes
curl -s -o /dev/null -w "anahtarsiz     : %{http_code}\n" http://127.0.0.1:9180/apisix/admin/routes

gercek anahtar : 200
ham sablon     : 401
anahtarsiz     : 401

10.3 Smoke test

curl -s -o /dev/null -H "X-API-KEY: $KEY" -X PUT \
  http://127.0.0.1:9180/apisix/admin/routes/g8-smoke \
  -H "Content-Type: application/json" \
  -d '{"uri":"/smoke","plugins":{"mocking":{"content_type":"text/plain","response_status":200,"response_example":"G8-GW-DEV-OK"}}}'
curl -s http://127.0.0.1:9080/smoke -w "\nstatus=%{http_code}\n"

G8-GW-DEV-OK
status=200

10.4 Prometheus

9091 is published in the compose file, and every published port deserves a test.

curl -s -o /dev/null -w "9091=%{http_code}\n" http://127.0.0.1:9091/apisix/prometheus/metrics
echo "metrics=$(curl -s http://127.0.0.1:9091/apisix/prometheus/metrics | grep -c '^apisix_')"

9091=200
metrics=87

10.5 Is the configuration persisted in etcd

sudo docker exec g8-etcd etcdctl get --prefix /apisix/routes --keys-only

/apisix/routes/
/apisix/routes/g8-smoke

10.6 Restart resilience

sudo docker compose restart
for i in $(seq 1 12); do
  [ "$(sudo docker inspect -f '{{.State.Health.Status}}' g8-apisix)" = "healthy" ] && break
  sleep 8
done
curl -s http://127.0.0.1:9080/smoke -w "\nstatus=%{http_code}\n"

G8-GW-DEV-OK
status=200

11. Observability

This environment gets its own Elasticsearch and Prometheus: request records go to Elasticsearch, request metrics to Prometheus. There is no shared stack — each environment holds its own data.

component	version	source
Elasticsearch	9.5.2	`harbor.g8.dev/infra/elasticsearch:9.5.2`
Prometheus	3.15.0	`harbor.g8.dev/infra/prometheus:v3.15.0`

Prometheus starts with --enable-feature=promql-extended-range-selectors: Analytics counts each bucket exactly with the anchored range selector, which Prometheus 3.15 takes only behind that flag. From Prometheus 3.16 the selector is on by default and the line can go; without it, Analytics shows its counts as estimates.

The added load was measured: together the two services take roughly 2 GB of RAM. On an 8 GB machine they sit comfortably beside APISIX and etcd; the measurement left 5.3 GB free.

Elasticsearch writes to the same disk as etcd

This document installs both on one machine. etcd is sensitive to fsync latency and Elasticsearch is a heavy writer. In an installation that needs low latency, move Elasticsearch to a separate disk or a separate machine.

11.1 Services

Two services are added to the services: block of docker-compose.yml:

/opt/g8-gw/docker-compose.yml (services altina eklenir)
  elasticsearch:
    image: harbor.g8.dev/infra/elasticsearch:9.5.2
    container_name: g8-elasticsearch
    restart: unless-stopped
    environment:
      - node.name=es-dev
      - cluster.name=g8-dev
      - discovery.type=single-node
      - bootstrap.memory_lock=true
      - ES_JAVA_OPTS=-Xms1g -Xmx1g
      - xpack.security.enabled=true
      - xpack.security.enrollment.enabled=false
      - xpack.security.http.ssl.enabled=false
      - xpack.security.transport.ssl.enabled=false
      - ELASTIC_PASSWORD=${G8_ES_PASSWORD}
      - ingest.geoip.downloader.enabled=false
    volumes:
      - /opt/g8-gw/es-data:/usr/share/elasticsearch/data
    ports:
      - "9200:9200"
    ulimits:
      memlock:
        soft: -1
        hard: -1
      nofile:
        soft: 65536
        hard: 65536
    healthcheck:
      test: ["CMD-SHELL", "curl -sf -u elastic:$${ELASTIC_PASSWORD} http://127.0.0.1:9200/_cluster/health | grep -qE '\"status\":\"(green|yellow)\"'"]
      interval: 15s
      timeout: 10s
      retries: 10
      start_period: 90s
    logging: *log-rotation
    networks: [g8gw]

  prometheus:
    image: harbor.g8.dev/infra/prometheus:v3.15.0
    container_name: g8-prometheus
    restart: unless-stopped
    command:
      - --config.file=/etc/prometheus/prometheus.yml
      - --storage.tsdb.path=/prometheus
      - --storage.tsdb.retention.time=40d
      - --storage.tsdb.retention.size=5GB
      - --enable-feature=promql-extended-range-selectors
    volumes:
      - /opt/g8-gw/prometheus/prometheus.yml:/etc/prometheus/prometheus.yml:ro
      - /opt/g8-gw/prom-data:/prometheus
    ports:
      - "9090:9090"
    healthcheck:
      test: ["CMD", "wget", "-qO-", "http://127.0.0.1:9090/-/healthy"]
      interval: 15s
      timeout: 5s
      retries: 5
      start_period: 20s
    logging: *log-rotation
    networks: [g8gw]

Data directories and the Elasticsearch password. The password is generated on the machine and written to neither this document nor the repository. It is the store administrator's password and no rule of the gateway carries it: the gateway writes with an account of its own (11.4).

sudo install -d -o 1000 -g 1000 -m 0755 /opt/g8-gw/es-data
sudo install -d -o 65534 -g 65534 -m 0755 /opt/g8-gw/prom-data
sudo install -d -m 0755 /opt/g8-gw/prometheus
printf 'G8_ES_PASSWORD=%s\n' "$(openssl rand -base64 24 | tr -d /+= | head -c 24)" \
  | sudo tee -a /opt/g8-gw/.env >/dev/null
sudo chmod 600 /opt/g8-gw/.env

The Prometheus configuration. external_labels marks the environment; when metrics from several environments meet, this is the only thing that tells them apart:

/opt/g8-gw/prometheus/prometheus.yml
global:
  scrape_interval: 15s
  evaluation_interval: 15s
  external_labels:
    g8_environment: dev

scrape_configs:
  - job_name: apisix
    metrics_path: /apisix/prometheus/metrics
    static_configs:
      - targets: ["apisix:9091"]
        labels:
          g8_environment: dev

  - job_name: prometheus
    static_configs:
      - targets: ["127.0.0.1:9090"]

11.2 Start

cd /opt/g8-gw
sudo docker compose up -d
sudo docker ps --format '{{.Names}} {{.Status}}'

g8-apisix Up 2 minutes (healthy)
g8-prometheus Up 2 minutes (healthy)
g8-elasticsearch Up 2 minutes (healthy)
g8-etcd Up 2 days (healthy)

Elasticsearch takes 60-90 seconds to report healthy; that is what start_period: 90s is for.

11.3 Wiring APISIX

The metrics are enabled through a global rule, which leaves prefer_name off: route and service on the metrics carry the ids Analytics counts by, and metrics.*.extra_labels in config.yaml put each route's and service's name beside them as route_name and service_name. The request records are not wired here: g8 writes their rule itself when traffic logging is turned on (11.5).

sudo tee /opt/g8-gw/wire-observability.sh >/dev/null <<'SCRIPT'
#!/bin/bash
set -eu
KEY=$(sudo grep '^G8_APISIX_ADMIN_KEY=' /opt/g8-gw/.env | cut -d= -f2-)
code=$(curl -s -o /dev/null -w '%{http_code}' -H "X-API-KEY: $KEY" \
  -X PUT http://127.0.0.1:9180/apisix/admin/global_rules/1 \
  -H 'Content-Type: application/json' -d '{"plugins":{"prometheus":{"prefer_name":false}}}')
echo "global_rule -> $code"
SCRIPT
sudo chmod 700 /opt/g8-gw/wire-observability.sh
sudo /opt/g8-gw/wire-observability.sh

global_rule -> 201

The prometheus plugin must be enabled; plugin_attr is not enough

plugin_attr.prometheus in config.yaml only sets the address the metrics are published on. Unless the plugin is enabled on a route or on a global rule, no request metrics are collected at all. The symptom is treacherous: the Prometheus target reads up, /apisix/prometheus/metrics returns 200, and yet a query for apisix_http_status comes back with zero series. That is the measured behaviour.

11.4 Accounts, the ingest pipeline and the index template in the log store

Unverified

The commands of this section were run on a development stack on 2026-10-02 (APISIX 3.19.0, Elasticsearch 8.15.3) with the outputs shown. On Elasticsearch 9.5.2 the privileges were measured with roles that name the days by a wildcard, and without the line that reads the template; the roles below name the days by a regular expression, which was measured on 8.15.3 alone. The commands have not been run on this machine yet.

The ingest pipeline, the line of the server's role that reads it and the move of 15.2 were run on 2026-10-04 against a development stack's Elasticsearch 8.15.3 with the outputs shown; that a batch sent twice is stored once was also measured in the lab on Elasticsearch 9.5.2 and OpenSearch 3.9.0.

Two accounts are opened in Elasticsearch, and neither of them is elastic:

account	used by	may
`g8-dev-gateway`	the gateway, to deliver the request records	write records into the daily indices of this environment; it reads no record, deletes none and sees nothing of the cluster
`g8-dev-server`	the g8 server, to show the records and to clean them up	read the state of the cluster and g8's ingest pipeline, read the daily indices of this environment and delete a day by its name

The daily indices carry the slug of the environment in g8 (g8-<slug>-access-<yyyy.mm.dd>), so ENV below must be the slug this gateway is registered under in the g8 console. The roles name the days by a regular expression and not by g8-dev-access-*: a wildcard would also reach the days of an environment whose slug begins with dev-access.

sudo tee /opt/g8-gw/store-accounts.sh >/dev/null <<'SCRIPT'
#!/bin/bash
set -eu
ENV=dev
ES=http://127.0.0.1:9200
for who in GATEWAY SERVER; do
  grep -q "^G8_ES_${who}_PASSWORD=" /opt/g8-gw/.env || \
    printf 'G8_ES_%s_PASSWORD=%s\n' "$who" "$(openssl rand -base64 24 | tr -d /+= | head -c 24)" >> /opt/g8-gw/.env
done
ESP=$(grep '^G8_ES_PASSWORD=' /opt/g8-gw/.env | cut -d= -f2-)
GWP=$(grep '^G8_ES_GATEWAY_PASSWORD=' /opt/g8-gw/.env | cut -d= -f2-)
SVP=$(grep '^G8_ES_SERVER_PASSWORD=' /opt/g8-gw/.env | cut -d= -f2-)
put() {
  code=$(curl -s -o /dev/null -w '%{http_code}' -u "elastic:$ESP" -X PUT "$ES$1" \
    -H 'Content-Type: application/json' --data-binary @-)
  echo "$1 -> $code"
}
DAYS="/g8-$ENV-access-[0-9]{4}[.][0-9]{2}[.][0-9]{2}/"
put /_security/role/g8-$ENV-gateway <<EOF
{"indices":[{"names":["$DAYS"],"privileges":["index","auto_configure"]}]}
EOF
put /_security/role/g8-$ENV-server <<EOF
{"cluster":["monitor","read_pipeline"],"indices":[{"names":["$DAYS"],
 "privileges":["read","view_index_metadata","monitor","delete_index"]}]}
EOF
put /_security/user/g8-$ENV-gateway <<EOF
{"password":"$GWP","roles":["g8-$ENV-gateway"]}
EOF
put /_security/user/g8-$ENV-server <<EOF
{"password":"$SVP","roles":["g8-$ENV-server"]}
EOF
SCRIPT
sudo chmod 700 /opt/g8-gw/store-accounts.sh
sudo /opt/g8-gw/store-accounts.sh

/_security/role/g8-dev-gateway -> 200
/_security/role/g8-dev-server -> 200
/_security/user/g8-dev-gateway -> 200
/_security/user/g8-dev-server -> 200

The script can be run again: it keeps the passwords it finds in /opt/g8-gw/.env.

The gateway's account must not be able to read

The password of g8-dev-gateway is part of the gateway's configuration: whoever can read the global rules of the gateway can obtain it. That is why this account can only add records. When traffic logging is turned on with a plain password, g8 tries the account and refuses one that can read the cluster or a record; a password given as a secret reference is resolved on the gateway and is not tried. create_doc is not enough for the account either: the plugin sends an index action, and with create_doc every record is refused while the request itself answers 200.

The ingest pipeline stores each record under its request id. The gateway's logger sends the records in batches and sends a batch again when the store answers it late, and by then the store has usually written the first one: under the request id, the second copy replaces the first instead of adding a record. The index template below sends every record through this pipeline, so the pipeline is applied first. A store that holds the template and not the pipeline refuses every record while the gateway's bulk request still answers 200; g8 checks that the store holds this version of the pipeline and does not turn traffic logging on without it. The store needs at least one node with the ingest role, which a node carries by default.

ESP=$(sudo grep '^G8_ES_PASSWORD=' /opt/g8-gw/.env | cut -d= -f2-)
curl -s -u "elastic:$ESP" -X PUT http://127.0.0.1:9200/_ingest/pipeline/g8-access \
  -H 'Content-Type: application/json' --data-binary @- <<'EOF'
{
  "description": "g8 request records: a record is keyed by its request id, so a delivery the gateway sends twice is stored once",
  "version": 1,
  "processors": [
    {
      "set": {
        "field": "_id",
        "value": "{{{request_id}}}",
        "if": "ctx.request_id instanceof String && ctx.request_id.length() == 32"
      }
    }
  ]
}
EOF

{"acknowledged":true}

The index template fixes the type of every field of a request record before the first index of a day is born. The store's administrator applies it once, with the command below. g8 checks that the store holds this version of it and does not turn traffic logging on without it.

ESP=$(sudo grep '^G8_ES_PASSWORD=' /opt/g8-gw/.env | cut -d= -f2-)
curl -s -u "elastic:$ESP" -X PUT http://127.0.0.1:9200/_index_template/g8-access \
  -H 'Content-Type: application/json' --data-binary @- <<'EOF'
{
  "index_patterns": [
    "g8-*-access-*"
  ],
  "priority": 200,
  "template": {
    "settings": {
      "index": {
        "number_of_shards": 1,
        "codec": "best_compression",
        "auto_expand_replicas": "0-1",
        "default_pipeline": "g8-access",
        "analysis": {
          "analyzer": {
            "g8_nodes": {
              "type": "pattern",
              "pattern": "(\\s*,\\s*)|(\\s+:\\s+)",
              "lowercase": false
            }
          }
        }
      }
    },
    "mappings": {
      "dynamic": false,
      "properties": {
        "start_time": {
          "type": "date",
          "format": "epoch_millis"
        },
        "latency": {
          "type": "float"
        },
        "upstream_latency": {
          "type": "float"
        },
        "apisix_latency": {
          "type": "float"
        },
        "client_ip": {
          "type": "ip",
          "ignore_malformed": true
        },
        "route_id": {
          "type": "keyword"
        },
        "service_id": {
          "type": "keyword"
        },
        "upstream": {
          "type": "keyword",
          "ignore_above": 1024,
          "fields": {
            "node": {
              "type": "text",
              "analyzer": "g8_nodes",
              "norms": false,
              "index_options": "docs"
            }
          }
        },
        "consumer": {
          "properties": {
            "username": {
              "type": "keyword"
            }
          }
        },
        "request": {
          "properties": {
            "method": {
              "type": "keyword"
            },
            "uri": {
              "type": "wildcard"
            },
            "url": {
              "type": "keyword",
              "index": false,
              "doc_values": false
            },
            "size": {
              "type": "long"
            },
            "headers": {
              "type": "object",
              "enabled": false
            },
            "querystring": {
              "type": "object",
              "enabled": false
            }
          }
        },
        "request_body": {
          "type": "text",
          "index": false
        },
        "response": {
          "properties": {
            "status": {
              "type": "short"
            },
            "size": {
              "type": "long"
            },
            "headers": {
              "type": "object",
              "enabled": false
            },
            "body": {
              "type": "text",
              "index": false
            }
          }
        },
        "server": {
          "properties": {
            "hostname": {
              "type": "keyword"
            },
            "version": {
              "type": "keyword"
            }
          }
        },
        "request_id": {
          "type": "keyword"
        },
        "route_name": {
          "type": "keyword"
        },
        "service_name": {
          "type": "keyword"
        },
        "upstream_status": {
          "type": "keyword"
        },
        "x_request_id": {
          "type": "keyword"
        },
        "logged_at": {
          "type": "date",
          "format": "epoch_second"
        }
      },
      "_meta": {
        "g8_records": 2
      },
      "_source": {
        "excludes": [
          "response.headers.set-cookie"
        ]
      }
    }
  }
}
EOF

{"acknowledged":true}

What the two accounts may and may not do, asked of the store itself. The day the check writes is deleted by its last line:

GWP=$(sudo grep '^G8_ES_GATEWAY_PASSWORD=' /opt/g8-gw/.env | cut -d= -f2-)
SVP=$(sudo grep '^G8_ES_SERVER_PASSWORD=' /opt/g8-gw/.env | cut -d= -f2-)
curl -s -o /dev/null -w 'g8-dev-gateway reads the cluster      -> %{http_code}\n' -u "g8-dev-gateway:$GWP" http://127.0.0.1:9200/
curl -s -o /dev/null -w 'g8-dev-gateway reads records          -> %{http_code}\n' -u "g8-dev-gateway:$GWP" 'http://127.0.0.1:9200/g8-dev-access-*/_search'
curl -s -o /dev/null -w 'g8-dev-gateway writes a day           -> %{http_code}\n' -u "g8-dev-gateway:$GWP" -X POST http://127.0.0.1:9200/g8-dev-access-2000.01.01/_doc -H 'Content-Type: application/json' -d '{}'
curl -s -o /dev/null -w 'g8-dev-gateway writes beside its days -> %{http_code}\n' -u "g8-dev-gateway:$GWP" -X POST http://127.0.0.1:9200/g8-dev-access-x-access-2000.01.01/_doc -H 'Content-Type: application/json' -d '{}'
curl -s -o /dev/null -w 'g8-dev-server reads the template      -> %{http_code}\n' -u "g8-dev-server:$SVP" http://127.0.0.1:9200/_index_template/g8-access
curl -s -o /dev/null -w 'g8-dev-server reads the pipeline      -> %{http_code}\n' -u "g8-dev-server:$SVP" http://127.0.0.1:9200/_ingest/pipeline/g8-access
curl -s -o /dev/null -w 'g8-dev-server reads beside its days   -> %{http_code}\n' -u "g8-dev-server:$SVP" 'http://127.0.0.1:9200/g8-dev-access-x-access-2000.01.01/_search'
curl -s -o /dev/null -w 'g8-dev-server writes a record         -> %{http_code}\n' -u "g8-dev-server:$SVP" -X POST http://127.0.0.1:9200/g8-dev-access-2000.01.01/_doc -H 'Content-Type: application/json' -d '{}'
curl -s -o /dev/null -w 'g8-dev-server deletes the day         -> %{http_code}\n' -u "g8-dev-server:$SVP" -X DELETE http://127.0.0.1:9200/g8-dev-access-2000.01.01

g8-dev-gateway reads the cluster      -> 403
g8-dev-gateway reads records          -> 403
g8-dev-gateway writes a day           -> 201
g8-dev-gateway writes beside its days -> 403
g8-dev-server reads the template      -> 200
g8-dev-server reads the pipeline      -> 200
g8-dev-server reads beside its days   -> 403
g8-dev-server writes a record         -> 403
g8-dev-server deletes the day         -> 200

Data streams cannot be used

elasticsearch-logger writes an index action in its bulk request; data streams require create and reject index. Hence one dated index per day.

11.5 Turning on the request records

Unverified

These steps were walked on a development stack on 2026-10-02, with the two accounts of 11.4. They have not been walked against this machine yet.

The request records are turned on from the g8 console, once this gateway is registered there as the environment dev. No command on this machine writes the logger's rule.

Environments › dev › Edit › Observability. Elasticsearch URL: the address the g8 server reaches this store at. Username g8-dev-server, password the value of G8_ES_SERVER_PASSWORD in /opt/g8-gw/.env. Test connection, then save.
Observe › Traffic Explorer › Settings. The Status card must read Log store Connected, Index template Applied and Ingest pipeline Applied.
Turn on traffic logging. Log store address, as the gateway reaches it: http://elasticsearch:9200. Username g8-dev-gateway, password the value of G8_ES_GATEWAY_PASSWORD in /opt/g8-gw/.env. Turn on.

g8 writes one global rule, g8-request-records, beside rule 1: the logger with the daily index of this environment, the fields the records need and the default masks. What is recorded and masked is changed afterwards on the same Settings page. Where the gateway writes is shown on the Status card as Gateway writes to; Change address gives the logger another address and account for the same store, through the same checks of the account as turning on.

11.6 Retention

A log store does not shrink on its own. With no limit the disk fills eventually, and past 95% Elasticsearch turns its indices read-only.

g8 deletes the request records: once a day it deletes, by its name, every daily index older than the number of days the environment keeps — 30 unless it is changed under Traffic Explorer › Settings › Retention › Change — and writes each deletion to the audit log. The store carries no lifecycle policy for these indices; a second deleter would delete days g8 still shows.

A day is named by its UTC date, by the gateway that writes it and by g8 that reads and deletes it.

On a single-node Elasticsearch a replica can never be allocated. The template sets auto_expand_replicas: 0-1: an index has no replica on one node and one as soon as a second node joins. The verification below must show 0 and green.

Prometheus is given two limits at once: --storage.tsdb.retention.time=40d and --storage.tsdb.retention.size=5GB. Whichever fills first does the cutting; a time limit alone does not keep the disk from filling. Forty days keep the whole 30-day window of Analytics.

11.7 Verification

Unverified

The output below was recorded on this machine while the rule of the earlier guide was in place; the index it shows is named by the machine's local day. With the rule g8 writes, the commands were run on a development stack on 2026-10-02: the index of the UTC day showed the requests sent, 0 and green. They have not been run on this machine since.

ESP=$(sudo grep '^G8_ES_PASSWORD=' /opt/g8-gw/.env | cut -d= -f2-)
for i in $(seq 1 25); do curl -s -o /dev/null http://127.0.0.1:9080/smoke; done
sleep 20
curl -s -u "elastic:$ESP" 'http://127.0.0.1:9200/_cat/indices/g8-*?v&h=index,docs.count,rep,health'
curl -s 'http://127.0.0.1:9090/api/v1/query?query=sum(apisix_http_status)'

index                    docs.count rep health
g8-dev-access-2026.08.29         25   0 green

The metric query must return the number of requests sent for apisix_http_status. The index must show 0 in the rep column and green in health.

Traffic Explorer in the g8 console must list the same requests.

Analytics reads two things from these metrics: each route under its id with its name beside it, and each bucket counted exactly with the anchored range selector:

curl -sG http://127.0.0.1:9090/api/v1/query --data-urlencode 'query=count by (route, route_name) (apisix_http_status)'
curl -sG http://127.0.0.1:9090/api/v1/query --data-urlencode 'query=sum(increase(apisix_http_status[5m] anchored))'

The first query lists every route that answered with its id in route and its name in route_name; a route that carries a name means a copy of the prometheus plugin has prefer_name on, and Analytics reads that route empty. The second must answer "status":"success"; a Prometheus started without --enable-feature=promql-extended-range-selectors answers anchored modifier is experimental and not enabled, and Analytics marks its counts as estimates. Both queries were run on this machine on 2026-10-04 against Prometheus 3.15.0: every route with a name showed it in route_name, and the anchored query answered success.

Request records are LOST while Elasticsearch is unreachable

The gateway sends a batch of records once more after the first failure (max_retry_count 1 in the rule g8 writes) and then drops it. The loss was measured with the rule of the earlier guide, which tried three times: on a production installation during a password change, 20 requests that arrived while Elasticsearch was restarting appeared in the metrics but never reached the index.

Read the loss from APISIX's error log, not from a counter difference:

grep -c 'elasticsearch-logger' <error.log>

Subtracting the document count from the metric counter is not reliable and produces false alarms. A measured example: counter 88, documents 43, difference 45 — with not one delivery error in the log. The difference has three causes and none of them is loss: the counter counts from the moment APISIX started (requests from before the plugin was enabled are in the counter but not the index), a restart resets the counter while the index persists, and the two sides have different retention. The difference is a hint only within a single window — no APISIX restart, nothing deleted by retention, both plugins enabled together.

An installation that cannot afford losing records must put a buffer in between (kafka-logger, or a local file-logger plus a separate shipper).

Restarts were measured to lose nothing: across a container restart and a full machine reboot the document count did not change, and Prometheus data from before the restart was still queryable. Persistence comes from the es-data and prom-data bind mounts; delete those two directories and the data is gone.

12. Diagnostics

When something does not work, look at these four in order:

cd /opt/g8-gw
sudo docker compose ps
sudo docker compose logs --tail 50 apisix
sudo tail -50 /opt/g8-gw/apisix/logs/error.log
sudo docker exec g8-etcd etcdctl endpoint health

APISIX's own error.log is separate from Docker's log driver; docker logs will not show the errors that appear there.

13. Backup and restore

etcd holds the entire gateway configuration. Without a backup, losing the machine means losing every route, upstream, consumer and plugin configuration.

sudo docker exec g8-etcd etcdctl snapshot save /etcd-data/snapshot.db
sudo mv /opt/g8-gw/etcd-data/snapshot.db /opt/g8-gw/backups/etcd-$(date +%Y%m%d-%H%M).db

Verify the snapshot's integrity — do not skip this; a corrupt backup is not a backup:

F=$(ls -t /opt/g8-gw/backups/*.db | head -1)
sudo docker run --rm -v /opt/g8-gw/backups:/b harbor.g8.dev/infra/etcd:v3.5.17 \
  etcdctl snapshot status /b/$(basename "$F") --write-out=table

Deprecated: Use `etcdutl snapshot status` instead.

+----------+----------+------------+------------+
|   HASH   | REVISION | TOTAL KEYS | TOTAL SIZE |
+----------+----------+------------+------------+
| adb4ac49 |      211 |         35 |      45 kB |
+----------+----------+------------+------------+

There is no automated backup here, deliberately

dev is a development environment; data durability is not a goal. This document installs no timer, so no backup is ever taken on its own. The command above is a one-off backup you run by hand.

If you are building an environment where the data must survive, do not use dev as the reference: the prod document's backup section describes the timer, the timeout, the pruning and the off-node copy together.

13.1 Restore

This procedure has been drilled on this environment. The drill used a marker pair: a route written before the backup came back after the restore, and one written after the backup did not. You can repeat the same drill yourself; nothing short of it shows that a backup is genuinely restorable.

Stop the services first — the data directory cannot be replaced while etcd runs:

cd /opt/g8-gw
sudo docker compose stop

Unpack the snapshot into a new directory; do not write over the existing one:

F=$(ls -t /opt/g8-gw/backups/*.db | head -1)
sudo rm -rf /opt/g8-gw/etcd-data-restored
sudo docker run --rm -v /opt/g8-gw/backups:/b -v /opt/g8-gw:/target \
  harbor.g8.dev/infra/etcd:v3.5.17 \
  etcdutl snapshot restore /b/$(basename "$F") \
    --name=etcd-dev \
    --initial-cluster=etcd-dev=http://etcd:2380 \
    --initial-advertise-peer-urls=http://etcd:2380 \
    --initial-cluster-token=g8-dev \
    --data-dir=/target/etcd-data-restored

The four flags must match the compose file exactly

--name, --initial-cluster, --initial-advertise-peer-urls and --initial-cluster-token write the cluster identity into the snapshot. If even one differs from the value in docker-compose.yml, etcd comes up as a member of a different cluster and APISIX cannot reach it. Do not guess the values, read them from the compose file.

Look at the directory that was produced:

sudo ls /opt/g8-gw/etcd-data-restored

member

--data-dir is the directory that contains member, not member itself

Passing .../etcd-data/member as --data-dir writes one level too deep. It succeeds silently; etcd then comes up with an empty store and you lose the very configuration you were restoring. The ls above must show exactly member.

Move the old one aside, put the new one in place, and bring things up:

sudo mv /opt/g8-gw/etcd-data /opt/g8-gw/etcd-data.old
sudo mv /opt/g8-gw/etcd-data-restored /opt/g8-gw/etcd-data
sudo docker compose up -d

Verify. The listing must show the configuration as it stood when the backup was taken:

KEY=$(sudo grep G8_APISIX_ADMIN_KEY /opt/g8-gw/.env | cut -d= -f2)
curl -s -H "X-API-KEY: $KEY" http://127.0.0.1:9180/apisix/admin/routes | grep -o '"id":"[^"]*"'

"id":"g8-smoke"

Everything written after the backup is gone; that is what a restore means. On the data plane, do not confuse 502 with 404: 502 means the route loaded but its upstream is unreachable, 404 means there is no such route at all.

Delete the old directory once verification passes — not before:

sudo rm -rf /opt/g8-gw/etcd-data.old

13.2 Defragmentation

Compaction removes old revisions logically but does not shrink the database file — the freed pages stay inside etcd and never return to the filesystem. Since the quota watches dbSize, the file keeps growing even when compaction is working perfectly, and the quota fills sooner or later. The extreme measured: dbSizeInUse at 41 kB while dbSize sat at 98 MB.

Defragmentation therefore has to be scheduled. Weekly is enough.

sudo tee /opt/g8-gw/etcd-defrag.sh >/dev/null <<'SCRIPT'
#!/bin/sh
set -eu
docker exec g8-etcd etcdctl --endpoints=http://127.0.0.1:2379 --command-timeout=900s defrag
docker exec g8-etcd etcdctl --endpoints=http://127.0.0.1:2379 --command-timeout=30s alarm list
SCRIPT
sudo chmod 0755 /opt/g8-gw/etcd-defrag.sh
sudo tee /etc/systemd/system/g8-etcd-defrag.service >/dev/null <<'EOF'
[Unit]
Description=g8 etcd defragmentation
After=docker.service
Requires=docker.service

[Service]
Type=oneshot
TimeoutStartSec=1800
ExecStart=/opt/g8-gw/etcd-defrag.sh
EOF

sudo tee /etc/systemd/system/g8-etcd-defrag.timer >/dev/null <<'EOF'
[Unit]
Description=g8 etcd defragmentation (weekly)

[Timer]
OnCalendar=Sun 03:00
Persistent=true

[Install]
WantedBy=timers.target
EOF
sudo systemctl daemon-reload
sudo systemctl enable --now g8-etcd-defrag.timer

Verification:

sudo systemctl start g8-etcd-defrag.service
systemctl show -p Result --value g8-etcd-defrag.service

success

Defrag blocks that member briefly, which is why the members are processed one at a time. Defragmenting all three at once drops quorum.

Defrag temporarily needs free space equal to the database size; that is one reason the quota must be meaningfully smaller than the disk.

14. Uninstall

Delete the environment in the g8 console first, or clear its log store there: g8 connects to the store once a day to delete the days past the retention, and records a failed cleanup every day for as long as the environment points at a store that is gone.

cd /opt/g8-gw && sudo docker compose down

To remove everything including the data — this deletes the etcd data:

cd /opt/g8-gw && sudo docker compose down -v
sudo rm -rf /opt/g8-gw/etcd-data /opt/g8-gw/apisix/logs

Everything else the installation left behind:

sudo rm -f /etc/logrotate.d/g8-apisix
sudo rm -rf /opt/g8-gw

Reverting the CA trust:

sudo rm -f /usr/local/share/ca-certificates/g8-lab-ca.crt
sudo rm -rf /etc/docker/certs.d/harbor.g8.dev
sudo update-ca-certificates --fresh

15. Upgrade

Unverified

On 2026-09-30 this environment's APISIX moved from 3.18.0 to 3.19.0 in place: the image tag in /opt/g8-gw/docker-compose.yml was changed (the previous file kept as docker-compose.yml.bak-<epoch>), docker compose up -d --no-deps apisix recreated the container, and the data plane answered Server: APISIX/3.19.0. That is a record of the change, not a drilled procedure with its verification. The next APISIX transition and etcd 3.5 → 3.6 have not been exercised in this environment. The procedure will be written after a real upgrade drill. Until then, keep the versions pinned.

15.1 Request records of an installation made with the earlier guide

Unverified

This move was rehearsed on a development stack on 2026-10-02: an installation made as the earlier guide wrote it, then the steps below in this order, with the outputs shown. It has not been carried out on this machine yet.

The earlier version of this guide wrote the logger into global rule 1 by hand: with the elastic account, an index named g8-dev-access- and the machine's local day, and a lifecycle policy that deleted the indices. Such an installation is moved in this order:

Do not turn traffic logging on before step 6

Settings offers Turn on traffic logging as soon as the pipeline and the template are applied. Turning on over the old rule completes it in place: the account is neither asked for nor tried, and elastic stays in the rule.

Open the two accounts (11.4).
Connect the store in the g8 console with g8-dev-server (step 1 of 11.5). Settings then reads Index template Outdated, Ingest pipeline Missing and the logger rule Customized. From its next daily cleanup on, g8 deletes the days past the retention.
Apply the ingest pipeline and then the index template (11.4). They apply to the days born after them.
Take the lifecycle policy off the indices and delete it:

ESP=$(sudo grep '^G8_ES_PASSWORD=' /opt/g8-gw/.env | cut -d= -f2-)
curl -s -u "elastic:$ESP" -X PUT 'http://127.0.0.1:9200/g8-*-access-*/_settings' \
  -H 'Content-Type: application/json' -d '{"index.lifecycle.name":null}'
curl -s -u "elastic:$ESP" -X DELETE http://127.0.0.1:9200/_ilm/policy/g8-access-retention

{"acknowledged":true}{"acknowledged":true}

The policy kept the indices 7 days. g8 keeps them 30 days unless that is changed: set the number of days this store is sized for under Traffic Explorer › Settings › Retention › Change.
Run the whole block of 11.3 again. It writes wire-observability.sh anew before it runs it: the file on this machine is the earlier guide's and would write the logger back. The block prints global_rule -> 200 here; rule 1 then carries the metrics alone and Settings reads the logger rule Off. Only then turn traffic logging on, with g8-dev-gateway (steps 2 and 3 of 11.5).
Change the password of elastic: it was part of the gateway's configuration for as long as the old rule existed.

No record is written between the two halves of step 6. Every day written before the move, and the day of the move itself, was born without g8's template and without the fields its records carry: Traffic Explorer cannot list those days, and Settings counts them as written under another version. They stay in the store until the retention deletes them. The records are listed from the first day whose index is born after the move.

g8 lists and deletes the days of the environment's own slug alone. If this gateway is registered in g8 under another slug than dev, the days the old rule wrote have no deleter once the policy is gone: delete them by their names when they are no longer needed.

15.2 The ingest pipeline on an installation made before it

Unverified

This move was rehearsed on 2026-10-04 against a development stack's Elasticsearch 8.15.3: an installation with the earlier template and a day born under it, then the steps below in this order, with the outputs shown. It has not been carried out on this machine yet.

An installation made before this version of the guide holds the index template without the ingest pipeline, and its server account may not read pipelines. Settings then reads Index template Outdated and Ingest pipeline Cannot be read. The gateway keeps writing records meanwhile, copies included, and the settings can still be saved; g8 refuses only to turn logging on anew until both read Applied. The move, in this order:

Run the whole block that writes the accounts script again (11.4). It rewrites the saved script with the new server role and runs it, keeping the passwords; every line answers 200. Running the saved script alone writes the old role again.
Apply the ingest pipeline (11.4).
Apply the index template (11.4). The days born after it send their records through the pipeline.
Send the day of today, born under the earlier template, through the pipeline at once. On a store that serves several environments, run it once for each environment's slug:

ESP=$(sudo grep '^G8_ES_PASSWORD=' /opt/g8-gw/.env | cut -d= -f2-)
curl -s -u "elastic:$ESP" -X PUT "http://127.0.0.1:9200/g8-dev-access-$(date -u +%Y.%m.%d)/_settings" \
  -H 'Content-Type: application/json' -d '{"index.default_pipeline":"g8-access"}'

{"acknowledged":true}

A store that holds no record of today yet answers 404 index_not_found_exception here: today's day will be born under the new template, and nothing more is needed. Settings then reads both Applied. A day written before the move may hold copies of a batch the gateway delivered twice; it stays as it is until the retention deletes it.

16. Traps hit during this install

Every item below actually happened during this install and was measured.

16.1 The APISIX image runs as uid 636

Symptom: the container restarts continuously and never reaches healthy.

nginx: [alert] could not open error log file:
open() "/usr/local/apisix/logs/error.log" failed (13: Permission denied)

Fix: sudo chown -R 636:636 /opt/g8-gw/apisix/logs

16.2 The image contains no HTTP client

Symptom: no error message at all. The container runs but stays unhealthy forever, and everything waiting on depends_on: service_healthy stalls.

The image contains no curl, wget, nc or busybox. It does contain bash:

test: ["CMD", "bash", "-c", "echo > /dev/tcp/127.0.0.1/9090"]

16.3 `/apisix/status` does not exist on 9080

Symptom:

{"error_msg":"404 Route Not Found"}

9080 is the data plane port; there is no status endpoint on it. The health check tests that the control API port 9090 is listening — the reason for not using 9080 is in 16.7.

16.4 Docker port publishing rewrites the source IP

Symptom: the Admin API returns 403 even with the correct key.

[error] access forbidden by rule, client: 172.18.0.1, request: "GET /apisix/admin/routes"

Requests arriving through a published ports: mapping appear to APISIX as coming from the bridge gateway. allow_admin must include that range.

The 192.0.2.0/24 entry in the allow_admin list in section 8 is this lab's network. Replace it with the network the g8 server sits on for a customer installation; otherwise even a request carrying the correct key gets a 403.

16.5 File descriptor limit

Symptom: under moderate load requests first slow down, then turn into 5xx with too many open files.

Warning! Current maximum number of open file descriptors [1024] is not greater than 1024

Fix: ulimits.nofile: 65536

16.6 Use `mocking`, not `echo`, for the smoke test

Symptom: the body is correct but the status is 503.

On a route with no upstream the echo plugin writes the body but APISIX still returns 503. Use mocking for a clean 200.

16.7 The health check fills the access log

Symptom: access.log grows even with no traffic at all; almost every line carries an empty uri and "status":"400".

sudo grep -c '"status":"400"' /opt/g8-gw/apisix/logs/access.log

echo > /dev/tcp/127.0.0.1/9080 opens a TCP connection and sends a newline. nginx treats that as a malformed request line, returns 400, and logs it. At a ten-second interval that is 8640 lines per node per day. The ratio measured in this environment before the fix: 1765 out of 1781 lines.

The control API (9090) does not write to the access log, so pointing the health check at it ends the pollution:

B=$(sudo wc -l < /opt/g8-gw/apisix/logs/access.log)
for i in $(seq 1 20); do sudo docker exec g8-apisix bash -c 'echo > /dev/tcp/127.0.0.1/9090'; done
sleep 2
echo "delta=$(( $(sudo wc -l < /opt/g8-gw/apisix/logs/access.log) - B ))"

delta=0

Do not treat delta=0 as evidence without running the negative control: the measuring tool may simply not have run. The negative control is the same loop aimed at 9080:

B=$(sudo wc -l < /opt/g8-gw/apisix/logs/access.log)
for i in $(seq 1 20); do sudo docker exec g8-apisix bash -c 'echo > /dev/tcp/127.0.0.1/9080'; done
sleep 8
echo "delta_9080=$(( $(sudo wc -l < /opt/g8-gw/apisix/logs/access.log) - B ))"

delta_9080=20

The sleep 8 is required; nginx flushes the access log with a delay and a two-second wait undercounts.

16.8 The `NOSPACE` alarm is a one-way door

Symptom: writes are refused with etcdserver: mvcc: database space exceeded. Nobody notices, because the data plane keeps returning 200 — and once it is noticed, freeing space is not enough.

The recovery chain was measured, and all three steps are required:

159 writes         -> dbSize=16789504  alarm=NOSPACE
delete + compact   -> dbSize=16789504  write refused
defrag             -> dbSize=122880    write refused
alarm disarm       -> write OK

There are three separate misconceptions here, each verified individually:

Deleting keys does not free space. etcd keeps the revision history; a delete writes a new revision.
Compaction does not return disk space. dbSizeInUse drops, dbSize does not move at all — the freed pages stay free inside etcd and never return to the filesystem. Since the quota watches dbSize, the quota is still full. The extreme measured in this environment: dbSizeInUse at 41 kB while dbSize sat at 98 MB.
Defrag returns the space but does not lower the alarm. The alarm is sticky: it survives a process restart, a container recreation and a host reboot.

Recovery:

sudo docker exec g8-etcd etcdctl --command-timeout=900s defrag
sudo docker exec g8-etcd etcdctl alarm list
sudo docker exec g8-etcd etcdctl alarm disarm
sudo docker exec g8-etcd etcdctl put /probe x

Leave alarm disarm for last: disarming without actually freeing space puts you straight back where you were on the next write.

The lasting fix is scheduled defragmentation — see long-term operation. With no defrag scheduled, dbSize keeps growing even when compaction is working correctly, and the quota fills sooner or later.

16.9 If the data directory is emptied, every check stays green

Symptom: none. That is precisely the problem.

If etcd loses its data directory (disk failure, accidental deletion, a wrong rm), then with ETCD_INITIAL_CLUSTER_STATE=new and an empty directory it bootstraps a fresh, empty cluster. The result:

endpoint health → true
both services active / healthy
all four ports listening
the smoke test… 404

The whole verification chain is green, because an empty cluster is a healthy cluster. It simply holds no configuration; every route the customer had is gone.

The only check that catches this asks about content, not health:

echo "routes=$(sudo docker exec g8-etcd etcdctl get /apisix/routes --prefix --keys-only | grep -c 'routes/g8')"

routes=1

If you see routes=0, there is no configuration regardless of what the services say. Recovery is a restore from backup — see restore. This environment takes no backup on its own: if the command in backup was never run by hand, there is nothing to restore.

17. Long-term operation

topic	setting	why
etcd growth	`--auto-compaction-mode=periodic`, `--auto-compaction-retention=1`	etcd keeps revision history indefinitely; without compaction every configuration write accumulates permanently
etcd quota	`--quota-backend-bytes=2147483648`	If the quota is exceeded etcd goes read-only. An early, explicit limit beats a silently full disk
Docker logs	`max-size: 10m`, `max-file: 3`	The default `json-file` driver grows without bound; months later what fills up is the host, not the container
APISIX access log	`logrotate` on the host + `docker kill -s USR1`	Docker's log driver does not see this file; rotation is set up separately (17.1)
Restart	`restart: unless-stopped` + etcd bind mount	Configuration survives container recreation (10.6)
Backup	Daily `etcdctl snapshot save`	Section 13

17.1 APISIX log rotation

Docker's max-size/max-file quota covers only the container's stdout. APISIX, however, writes access.log and error.log as files into the bind-mounted directory; those files fall outside the quota and grow without bound. The rate measured in this environment is roughly 140 bytes per request.

After rotation nginx is sent USR1; it reopens the files and no line is lost. PID 1 in the container is the nginx master process, so the signal reaches it via docker kill -s USR1.

sudo tee /etc/logrotate.d/g8-apisix >/dev/null <<'EOF'
/opt/g8-gw/apisix/logs/*.log {
    daily
    maxsize 50M
    rotate 14
    missingok
    notifempty
    compress
    delaycompress
    sharedscripts
    postrotate
        /usr/bin/docker kill -s USR1 g8-apisix >/dev/null 2>&1 || true
    endscript
}
EOF

That rotation is lossless is measured. The count targets only /smoke lines; counting total lines is unstable because of nginx's delayed flush.

sudo truncate -s 0 /opt/g8-gw/apisix/logs/access.log
for i in 1 2 3; do curl -s -o /dev/null http://127.0.0.1:9080/smoke; done
sleep 3
sudo logrotate -f /etc/logrotate.d/g8-apisix
sleep 3
for i in 1 2 3 4 5; do curl -s -o /dev/null http://127.0.0.1:9080/smoke; done
sleep 3
echo "new=$(sudo grep -c '"uri":"/smoke"' /opt/g8-gw/apisix/logs/access.log)"
echo "old=$(sudo grep -c '"uri":"/smoke"' /opt/g8-gw/apisix/logs/access.log.1)"

new=5
old=3

Write maxsize, not size

With size 50M, logrotate abandons time-based rotation entirely; the file never rotates until it exceeds 50 MB.

18. Scope limits and next steps

limit	where it is addressed
Fault tolerance — single node	Prod install (3 APISIX + 3 etcd)
APISIX ↔ etcd traffic is unencrypted	TLS document
Behaviour under load not measured	Load test document
Upgrade path not exercised	Section 15

Once this environment is running, the next step is registering the cluster as an environment in the g8 console.

G8 dev · Apache APISIX® 3.19.0

Apache, Apache APISIX and APISIX are trademarks of The Apache Software Foundation.

1. What gets installed​

2. Prerequisites​

2.1 Ports​

3. Install Docker​

4. Trust the Harbor CA​

5. Directories​

6. Images​

7. Admin key​

8. Configuration files​

9. Start​

10. Verification​

10.1 Health​

10.2 Admin API and key substitution​

10.3 Smoke test​

10.4 Prometheus​

10.5 Is the configuration persisted in etcd​

10.6 Restart resilience​

11. Observability​

11.1 Services​

11.2 Start​

11.3 Wiring APISIX​

11.4 Accounts, the ingest pipeline and the index template in the log store​

11.5 Turning on the request records​

11.6 Retention​

11.7 Verification​

12. Diagnostics​

13. Backup and restore​

13.1 Restore​

13.2 Defragmentation​

14. Uninstall​

15. Upgrade​

15.1 Request records of an installation made with the earlier guide​

15.2 The ingest pipeline on an installation made before it​

16. Traps hit during this install​

16.1 The APISIX image runs as uid 636​

16.2 The image contains no HTTP client​

16.3 /apisix/status does not exist on 9080​

16.4 Docker port publishing rewrites the source IP​

16.5 File descriptor limit​

16.6 Use mocking, not echo, for the smoke test​

16.7 The health check fills the access log​

16.8 The NOSPACE alarm is a one-way door​

16.9 If the data directory is emptied, every check stays green​

17. Long-term operation​

17.1 APISIX log rotation​

18. Scope limits and next steps​

1. What gets installed

2. Prerequisites

2.1 Ports

3. Install Docker

4. Trust the Harbor CA

5. Directories

6. Images

7. Admin key

8. Configuration files

9. Start

10. Verification

10.1 Health

10.2 Admin API and key substitution

10.3 Smoke test

10.4 Prometheus

10.5 Is the configuration persisted in etcd

10.6 Restart resilience

11. Observability

11.1 Services

11.2 Start

11.3 Wiring APISIX

11.4 Accounts, the ingest pipeline and the index template in the log store

11.5 Turning on the request records

11.6 Retention

11.7 Verification

12. Diagnostics

13. Backup and restore

13.1 Restore

13.2 Defragmentation

14. Uninstall

15. Upgrade

15.1 Request records of an installation made with the earlier guide

15.2 The ingest pipeline on an installation made before it

16. Traps hit during this install

16.1 The APISIX image runs as uid 636

16.2 The image contains no HTTP client

16.3 `/apisix/status` does not exist on 9080

16.4 Docker port publishing rewrites the source IP

16.5 File descriptor limit

16.6 Use `mocking`, not `echo`, for the smoke test

16.7 The health check fills the access log

16.8 The `NOSPACE` alarm is a one-way door

16.9 If the data directory is emptied, every check stays green

17. Long-term operation

17.1 APISIX log rotation

18. Scope limits and next steps