Prod environment — 3 APISIX + 3 etcd on Docker
The host names, addresses and the registry on this page are examples. Our lab run used its own values in their place; replace them with yours, and take the registry address and its credentials from your G8 contact.
Three APISIX nodes and a three-member etcd cluster, with Docker Compose. Configuration is written to any one APISIX node and propagates to the other two through etcd.
This document runs six containers on a single machine. That verifies the
shape of the configuration: peer URLs, initial-cluster, the quorum arithmetic,
the behaviour on member loss. When the host dies, all three members go with it
and there is no quorum left to speak of.
Real fault tolerance needs three separate machines. The compose file in this
document is moved there by splitting the etcd-* and apisix-* services across
three hosts and replacing the initial-cluster addresses with those hosts'
addresses. That move was not attempted in this environment.
1. What we are installing
| component | version | source |
|---|---|---|
| APISIX | 3.19.0-debian | harbor.g8.dev/infra/apisix:3.19.0-debian |
| etcd | v3.5.17 | harbor.g8.dev/infra/etcd:v3.5.17 |
The images are pulled from our own Harbor registry. The customer environment may be air-gapped; we leave no dependency on upstream (Docker Hub, quay.io).
| service | container | role |
|---|---|---|
etcd-0 etcd-1 etcd-2 | g8-etcd-0..2 | Configuration store, three-member raft cluster |
apisix-0 apisix-1 apisix-2 | g8-apisix-0..2 | Data plane; all three read the same config.yaml |
2. Prerequisites
| requirement | value | note |
|---|---|---|
| Operating system | Ubuntu 24.04 LTS | The CA trust command differs on other distributions |
| CPU / RAM / disk | 4 vCPU / 8 GB / 100 GB | Six containers; etcd is sensitive to fsync, SSD/NVMe recommended |
| Clock synchronisation | chrony or systemd-timesyncd active | etcd enters leader election on clock drift |
| Harbor access | harbor.g8.dev:443 | Image source |
| DNS | harbor.g8.dev must resolve | |
| Harbor CA file | g8-lab-ca.crt in the working directory | Shipped with the installation package; the CA step below copies this file |
| Internet access | download.docker.com, archive.ubuntu.com | Required. APISIX and etcd come from Harbor, but Docker and the OS packages are installed from upstream. A genuinely air-gapped site needs a local mirror for those steps |
2.1 Ports
| port | component | who should reach it |
|---|---|---|
| 9080 9081 9082 | APISIX data plane (node 0/1/2) | Clients; a load balancer goes in front |
| 9180 9181 9182 | APISIX Admin API | Bound to 127.0.0.1 only. Not exposed to the internet |
| 9091 9092 9093 | APISIX Prometheus | 127.0.0.1 only; if Prometheus is elsewhere, reach it over a tunnel or network |
| 9200 | Elasticsearch | The g8 server and the edge only. Authenticated, but there is no TLS — the password crosses the network in the clear |
| 9090 | Prometheus | The g8 server and the edge only. No authentication |
| 2379 2380 | etcd client and peer | Never published to the host; only inside the g8-gw Docker network |
The Admin API and Prometheus ports bind to 127.0.0.1: the verification commands
run from the machine itself and nothing is reachable from outside. The data plane
ports are published separately for all three nodes so each node can be tested on
its own; in production clients face a load balancer.
3. Installing Docker
On a freshly installed Ubuntu cloud image, unattended-upgrades holds the apt
locks on first boot and the apt-get install below fails silently — it does
return an exit code, but if that is swallowed in automation the install never
happened.
In automation, wait on all four locks before installing. Do not try to wait by
process name (pgrep unattended-upgrade):
unattended-upgrade-shutdown --wait-for-signal runs permanently and would lock
the loop forever.
for i in $(seq 1 90); do
BUSY=0
for L in /var/lib/dpkg/lock-frontend /var/lib/dpkg/lock \
/var/lib/apt/lists/lock /var/cache/apt/archives/lock; do
sudo fuser "$L" >/dev/null 2>&1 && BUSY=1
done
[ "$BUSY" = "0" ] && break
sleep 10
done
sudo apt-get update -qq
sudo apt-get install -y ca-certificates curl gnupg jq
sudo install -m 0755 -d /etc/apt/keyrings
curl -fsSL https://download.docker.com/linux/ubuntu/gpg | sudo gpg --dearmor -o /etc/apt/keyrings/docker.gpg
sudo chmod a+r /etc/apt/keyrings/docker.gpg
echo "deb [arch=amd64 signed-by=/etc/apt/keyrings/docker.gpg] https://download.docker.com/linux/ubuntu noble stable" \
| sudo tee /etc/apt/sources.list.d/docker.list >/dev/null
sudo apt-get update -qq
sudo apt-get install -y docker-ce docker-ce-cli containerd.io docker-buildx-plugin docker-compose-plugin
sudo systemctl enable --now docker
4. Trust the Harbor CA
Harbor presents a certificate signed by our own CA. No image can be pulled until
that CA is trusted — you get x509: certificate signed by unknown authority.
sudo cp g8-lab-ca.crt /usr/local/share/ca-certificates/g8-lab-ca.crt
sudo update-ca-certificates
sudo mkdir -p /etc/docker/certs.d/harbor.g8.dev
sudo cp g8-lab-ca.crt /etc/docker/certs.d/harbor.g8.dev/ca.crt
Verification — you must see sslverify=0:
curl -s -o /dev/null -w "sslverify=%{ssl_verify_result} status=%{http_code}\n" \
https://harbor.g8.dev/api/v2.0/systeminfo
sslverify=0 status=200
5. Directories
Each node has its own data and log directory. The APISIX image runs as uid 636 and, in this installation, the etcd containers run as uid 1000; directory ownership follows.
sudo mkdir -p /opt/g8-gw/apisix /opt/g8-gw/backups
for n in 0 1 2; do sudo mkdir -p /opt/g8-gw/etcd-data-$n /opt/g8-gw/apisix/logs-$n; done
sudo chown -R $(id -u):$(id -g) /opt/g8-gw
sudo chown -R 636:636 /opt/g8-gw/apisix/logs-0 /opt/g8-gw/apisix/logs-1 /opt/g8-gw/apisix/logs-2
sudo chmod 700 /opt/g8-gw/etcd-data-0 /opt/g8-gw/etcd-data-1 /opt/g8-gw/etcd-data-2
This document assumes id -u returns 1000; the user: "1000:1000" line in
the compose file must match it. Leaving the assumption in prose is not enough —
measure it:
echo "uid=$(id -u)"
test "$(id -u)" = 1000 && echo "match=yes" || echo "match=no"
uid=1000
match=yes
If a different number comes back, change the user: line in the compose file and
every other place that names that id.
6. Images
sudo docker pull harbor.g8.dev/infra/etcd:v3.5.17
sudo docker pull harbor.g8.dev/infra/apisix:3.19.0-debian
7. Admin key
APISIX's documented default admin key (edd1c9f0…) is public knowledge. Left
unchanged, the Admin API is wide open with full privileges.
The key lives in a .env file and is passed to all three containers as an
environment variable; it is never written by hand into config.yaml.
printf 'G8_APISIX_ADMIN_KEY=%s\n' "$(openssl rand -hex 24)" | sudo tee /opt/g8-gw/.env >/dev/null
sudo chmod 600 /opt/g8-gw/.env
8. APISIX configuration
A single config.yaml is mounted read-only into all three containers; the file
carries no node identity. The ${{G8_APISIX_ADMIN_KEY}} syntax belongs to
APISIX: it reads the value from the environment at startup.
etcd.host lists all three members — if one goes down, APISIX moves to another.
sudo tee /opt/g8-gw/apisix/config.yaml >/dev/null <<'EOF'
apisix:
node_listen:
- 9080
deployment:
role: traditional
role_traditional:
config_provider: etcd
admin:
admin_listen:
ip: 0.0.0.0
port: 9180
admin_key:
- name: admin
key: "${{G8_APISIX_ADMIN_KEY}}"
role: admin
allow_admin:
- 172.16.0.0/12
- 127.0.0.0/8
etcd:
host:
- "http://etcd-0:2379"
- "http://etcd-1:2379"
- "http://etcd-2:2379"
prefix: /apisix
timeout: 30
nginx_config:
error_log_level: warn
http:
access_log: /usr/local/apisix/logs/access.log
access_log_format: '{"time":"$time_iso8601","remote_addr":"$remote_addr","status":"$status","method":"$request_method","uri":"$uri","rt":"$request_time","ua":"$http_user_agent"}'
access_log_format_escape: json
plugin_attr:
prometheus:
export_addr:
ip: 0.0.0.0
port: 9091
metrics:
http_status:
extra_labels:
- route_name: $route_name
- service_name: $service_name
http_latency:
extra_labels:
- route_name: $route_name
- service_name: $service_name
EOF
allow_admin contains 172.16.0.0/12 because Docker port publishing rewrites
the source IP to the bridge gateway; a request from 127.0.0.1 reaches APISIX as
172.x.
9. The compose file
YAML anchors (x-…) keep the part shared by the six services in one place. The
heartbeat and election-timeout values are for a single host; they must be
reviewed against network latency when moving to separate machines.
sudo tee /opt/g8-gw/docker-compose.yml >/dev/null <<'EOF'
name: g8-gw
x-logging: &logging
driver: json-file
options:
max-size: "10m"
max-file: "3"
x-etcd-env: &etcd-env
ETCD_DATA_DIR: /etcd-data
ETCD_LISTEN_PEER_URLS: http://0.0.0.0:2380
ETCD_LISTEN_CLIENT_URLS: http://0.0.0.0:2379
ETCD_INITIAL_CLUSTER: etcd-0=http://etcd-0:2380,etcd-1=http://etcd-1:2380,etcd-2=http://etcd-2:2380
ETCD_INITIAL_CLUSTER_STATE: new
ETCD_INITIAL_CLUSTER_TOKEN: g8-prod
ETCD_AUTO_COMPACTION_MODE: periodic
ETCD_AUTO_COMPACTION_RETENTION: "1"
ETCD_QUOTA_BACKEND_BYTES: "2147483648"
ETCD_SNAPSHOT_COUNT: "10000"
ETCD_HEARTBEAT_INTERVAL: "250"
ETCD_ELECTION_TIMEOUT: "2500"
x-etcd: &etcd
image: harbor.g8.dev/infra/etcd:v3.5.17
restart: unless-stopped
user: "1000:1000"
networks: [g8]
logging: *logging
ulimits:
nofile:
soft: 65536
hard: 65536
healthcheck:
test: ["CMD", "etcdctl", "endpoint", "health"]
interval: 10s
timeout: 5s
retries: 6
start_period: 20s
x-apisix: &apisix
image: harbor.g8.dev/infra/apisix:3.19.0-debian
restart: unless-stopped
networks: [g8]
logging: *logging
env_file:
- .env
ulimits:
nofile:
soft: 65536
hard: 65536
healthcheck:
test: ["CMD", "bash", "-c", "echo > /dev/tcp/127.0.0.1/9090"]
interval: 10s
timeout: 5s
retries: 6
start_period: 20s
depends_on:
etcd-0:
condition: service_healthy
etcd-1:
condition: service_healthy
etcd-2:
condition: service_healthy
services:
etcd-0:
<<: *etcd
container_name: g8-etcd-0
environment:
<<: *etcd-env
ETCD_NAME: etcd-0
ETCD_ADVERTISE_CLIENT_URLS: http://etcd-0:2379
ETCD_INITIAL_ADVERTISE_PEER_URLS: http://etcd-0:2380
volumes:
- ./etcd-data-0:/etcd-data
etcd-1:
<<: *etcd
container_name: g8-etcd-1
environment:
<<: *etcd-env
ETCD_NAME: etcd-1
ETCD_ADVERTISE_CLIENT_URLS: http://etcd-1:2379
ETCD_INITIAL_ADVERTISE_PEER_URLS: http://etcd-1:2380
volumes:
- ./etcd-data-1:/etcd-data
etcd-2:
<<: *etcd
container_name: g8-etcd-2
environment:
<<: *etcd-env
ETCD_NAME: etcd-2
ETCD_ADVERTISE_CLIENT_URLS: http://etcd-2:2379
ETCD_INITIAL_ADVERTISE_PEER_URLS: http://etcd-2:2380
volumes:
- ./etcd-data-2:/etcd-data
apisix-0:
<<: *apisix
container_name: g8-apisix-0
ports:
- "0.0.0.0:9080:9080"
- "127.0.0.1:9180:9180"
- "127.0.0.1:9091:9091"
volumes:
- ./apisix/config.yaml:/usr/local/apisix/conf/config.yaml:ro
- ./apisix/logs-0:/usr/local/apisix/logs
apisix-1:
<<: *apisix
container_name: g8-apisix-1
ports:
- "0.0.0.0:9081:9080"
- "127.0.0.1:9181:9180"
- "127.0.0.1:9092:9091"
volumes:
- ./apisix/config.yaml:/usr/local/apisix/conf/config.yaml:ro
- ./apisix/logs-1:/usr/local/apisix/logs
apisix-2:
<<: *apisix
container_name: g8-apisix-2
ports:
- "0.0.0.0:9082:9080"
- "127.0.0.1:9182:9180"
- "127.0.0.1:9093:9091"
volumes:
- ./apisix/config.yaml:/usr/local/apisix/conf/config.yaml:ro
- ./apisix/logs-2:/usr/local/apisix/logs
networks:
g8:
name: g8-gw
EOF
Syntax check — it must print six services:
cd /opt/g8-gw && sudo docker compose config --services | sort | paste -sd,
apisix-0,apisix-1,apisix-2,etcd-0,etcd-1,etcd-2
10. Log rotation
Docker's max-size/max-file quota covers only the container's stdout. APISIX,
however, writes access.log and error.log as files into the bind-mounted
directories; those files fall outside the quota and grow without bound — details
in 18.4.
After rotation each APISIX container is sent USR1. PID 1 in the container is
the nginx master process, so the signal reaches it via docker kill -s USR1.
sudo tee /etc/logrotate.d/g8-apisix >/dev/null <<'EOF'
/opt/g8-gw/apisix/logs-*/*.log {
daily
maxsize 50M
rotate 14
missingok
notifempty
compress
delaycompress
sharedscripts
postrotate
for c in g8-apisix-0 g8-apisix-1 g8-apisix-2; do
/usr/bin/docker kill -s USR1 "$c" >/dev/null 2>&1 || true
done
endscript
}
EOF
Syntax check. The output is searched with grep, not sliced with head:
logrotate 3.21.0 prints a debug banner and state-file chatter first, pushing the
lines you want into the middle of the list — position 36 in this environment, and
somewhere else on another machine (18.7).
sudo logrotate -d /etc/logrotate.d/g8-apisix 2>&1 | grep -A1 'rotating pattern'
rotating pattern: /opt/g8-gw/apisix/logs-*/*.log after 1 days (14 rotations)
empty log files are not rotated, log files >= 52428800 are rotated earlier, old logs are removed
maxsize, not sizeWith size 50M, logrotate abandons time-based rotation entirely; the file
never rotates until it exceeds 50 MB.
11. Starting
The etcd cluster comes up first; the APISIX services wait via depends_on for
all three members to be healthy.
cd /opt/g8-gw
sudo docker compose up -d
S=$(sudo docker compose ps --services | wc -l)
for i in $(seq 1 25); do
[ "$(sudo docker compose ps --format '{{.Status}}' | grep -c '(healthy)')" = "$S" ] && break
sleep 6
done
sudo docker compose ps --format '{{.Name}} {{.Status}}' | sort
echo "healthy=$(sudo docker compose ps --format '{{.Status}}' | grep -c '(healthy)')/$S unhealthy=$(sudo docker compose ps --format '{{.Status}}' | grep -c unhealthy)"
healthy=8/8 unhealthy=0
restart: unless-stopped brings a container back when it dies, but when
Docker itself restarts the policy was measured not to fire: in one of four
attempts two APISIX containers stayed Exited (0) and two ports were dead for
223 seconds. Since nothing repairs the stack when the policy does not fire, a
systemd unit is needed:
sudo tee /etc/systemd/system/g8-gateway.service >/dev/null <<'EOF'
[Unit]
Description=g8 gateway stack (docker compose)
After=docker.service
Requires=docker.service
[Service]
Type=oneshot
RemainAfterExit=yes
WorkingDirectory=/opt/g8-gw
ExecStart=/usr/bin/docker compose up -d
ExecStop=/usr/bin/docker compose stop
TimeoutStartSec=600
[Install]
WantedBy=multi-user.target
EOF
sudo systemctl daemon-reload
sudo systemctl enable g8-gateway.service
12. Verification
12.1 Containers and cluster
The member count, the healthy-endpoint count and the leader count are each counted separately. A single "the cluster looks fine" glance is not enough: two leaders, or a learner member, also look like "running".
cd /opt/g8-gw
E=http://etcd-0:2379,http://etcd-1:2379,http://etcd-2:2379
etcdq() { sudo docker run --rm --network g8-gw --user 1000:1000 \
harbor.g8.dev/infra/etcd:v3.5.17 etcdctl --endpoints=$E "$@"; }
echo "healthy=$(sudo docker compose ps --format '{{.Status}}' | grep -c '(healthy)')/$(sudo docker compose ps --services | wc -l) unhealthy=$(sudo docker compose ps --format '{{.Status}}' | grep -c unhealthy)"
echo "members=$(etcdq member list -w simple | wc -l)"
echo "healthy_endpoints=$(etcdq endpoint health -w simple 2>&1 | grep -c 'is healthy')"
echo "leaders=$(etcdq endpoint status -w simple | awk -F, '$5 ~ /true/' | wc -l)"
echo "learners=$(etcdq endpoint status -w simple | awk -F, '$6 ~ /true/' | wc -l)"
healthy=8/8 unhealthy=0
members=3
healthy_endpoints=3
leaders=1
learners=0
Whether the file descriptor limit was actually applied is read from the running process, not from the definition:
sudo docker exec g8-apisix-0 sh -c 'ulimit -n'
sudo awk '/Max open files/{print $4}' /proc/$(sudo docker inspect -f '{{.State.Pid}}' g8-etcd-0)/limits
65536
65536
docker exec … sh is not used for etcd; the image contains no shell
(18.6).
12.2 Admin API and key substitution
The lab's prod host differs from this document in one place: apisix-0's Admin API is also published on the host as 192.0.2.108:9180, and its allow_admin list was widened to the lab segment (192.0.2.0/24), so that g8 on the OKD cluster can manage this environment. This document keeps the Admin API on loopback, which is the customer default; open it only for a management host and only to its address.
Three requests: the real key, the unsubstituted raw template, and no key. If the
second does not return 401, ${{...}} substitution did not happen and the key
was taken as literal text.
KEY=$(sudo grep '^G8_APISIX_ADMIN_KEY=' /opt/g8-gw/.env | cut -d= -f2-)
umask 077
printf 'header = "X-API-KEY: %s"\n' "$KEY" > /tmp/g8adm.conf
printf 'header = "X-API-KEY: ${{G8_APISIX_ADMIN_KEY}}"\n' > /tmp/g8raw.conf
curl -s -o /dev/null -w "real=%{http_code}\n" -K /tmp/g8adm.conf http://127.0.0.1:9180/apisix/admin/routes
curl -s -o /dev/null -w "raw=%{http_code}\n" -K /tmp/g8raw.conf http://127.0.0.1:9180/apisix/admin/routes
curl -s -o /dev/null -w "none=%{http_code}\n" http://127.0.0.1:9180/apisix/admin/routes
real=200
raw=401
none=401
The key is never written on the command line; curl -K reads it from a file. A
secret placed on the command line is visible in ps output and in shell history.
12.3 Does the configuration propagate to all three nodes
The real exam for this installation: the route is written to node 0 only and read back from all three.
curl -s -o /dev/null -w "PUT=%{http_code}\n" -K /tmp/g8adm.conf \
-X PUT http://127.0.0.1:9180/apisix/admin/routes/g8-smoke \
-H "Content-Type: application/json" \
-d '{"uri":"/smoke","plugins":{"mocking":{"content_type":"text/plain","response_status":200,"response_example":"G8-GW-PROD-OK"}}}'
sleep 3
for p in 9080 9081 9082; do
echo "$p=$(curl -s -o /dev/null -w '%{http_code}' http://127.0.0.1:$p/smoke) $(curl -s http://127.0.0.1:$p/smoke)"
done
PUT=201
9080=200 G8-GW-PROD-OK
9081=200 G8-GW-PROD-OK
9082=200 G8-GW-PROD-OK
12.4 Quorum: one member down
The leader is taken down deliberately; the cluster must elect a new one and keep accepting writes (with three members, quorum is two).
Cluster queries go through etcdq, a one-off container that depends on no
member at all. Using docker exec g8-etcd-0 would mean measuring through the
very node the drill is about to stop (18.7).
cd /opt/g8-gw
E=http://etcd-0:2379,http://etcd-1:2379,http://etcd-2:2379
etcdq() { sudo docker run --rm --network g8-gw --user 1000:1000 \
harbor.g8.dev/infra/etcd:v3.5.17 etcdctl --endpoints=$E "$@"; }
LEADER=$(etcdq endpoint status -w simple | awk -F, '$5 ~ /true/ {print $1}' | sed 's|http://||;s|:2379||')
echo "leader=$LEADER"
sudo docker compose stop "$LEADER"
sleep 12
echo "healthy_endpoints=$(etcdq endpoint health -w simple 2>&1 | grep -c 'is healthy')"
curl -s -o /dev/null --max-time 30 -w "PUT=%{http_code}\n" -K /tmp/g8adm.conf \
-X PUT http://127.0.0.1:9180/apisix/admin/routes/g8-quorum \
-H "Content-Type: application/json" \
-d '{"uri":"/quorum","plugins":{"mocking":{"content_type":"text/plain","response_status":200,"response_example":"QUORUM-OK"}}}'
sleep 3
echo "9081=$(curl -s -o /dev/null -w '%{http_code}' http://127.0.0.1:9081/quorum) $(curl -s http://127.0.0.1:9081/quorum)"
leader=etcd-0
healthy_endpoints=2
PUT=201
9081=200 QUORUM-OK
The leader line changes on every run — which member is leader is incidental, and
no step here writes a fixed name. What must not change is
healthy_endpoints=2 and PUT=201.
12.5 Quorum loss: two members down
Take a second member down and there is no quorum left. The expected behaviour has two parts and both must be measured: the write fails, while the data plane keeps serving existing routes from cache.
The second member to stop is picked from the running ones. Writing a fixed name means that if 12.4 already stopped it, this step silently does nothing and the drill appears to "pass" without ever losing quorum.
cd /opt/g8-gw
E=http://etcd-0:2379,http://etcd-1:2379,http://etcd-2:2379
etcdq() { sudo docker run --rm --network g8-gw --user 1000:1000 \
harbor.g8.dev/infra/etcd:v3.5.17 etcdctl --endpoints=$E "$@"; }
SECOND=$(sudo docker compose ps --services --filter status=running | grep '^etcd-' | head -1)
echo "second=$SECOND"
sudo docker compose stop "$SECOND"
sleep 12
echo "healthy_endpoints=$(etcdq endpoint health -w simple 2>&1 | grep -c 'is healthy')"
curl -s -o /dev/null --max-time 150 -w "PUT=%{http_code}\n" -K /tmp/g8adm.conf \
-X PUT http://127.0.0.1:9180/apisix/admin/routes/g8-noquorum \
-H "Content-Type: application/json" \
-d '{"uri":"/noquorum","plugins":{"mocking":{"content_type":"text/plain","response_status":200,"response_example":"NO"}}}'
for p in 9080 9081 9082; do
echo "$p=$(curl -s -o /dev/null --max-time 10 -w '%{http_code}' http://127.0.0.1:$p/smoke)"
done
second=etcd-1
healthy_endpoints=0
PUT=503
9080=200
9081=200
9082=200
Note healthy_endpoints=0: the surviving member is running but does not
count as healthy. endpoint health performs a read that requires quorum, and
with no quorum no member can serve it. So the 0 here says the cluster is
unwritable, not that a member is dead.
The write took 70–110 seconds to come back in this environment; the Admin API does not fail fast, with the detail and the measurements in 18.2. The body it returns varies per attempt, which is why no fixed error string is printed here.
The cluster is brought back. The start command names all three members: which one was stopped depends on the run, and starting an already-running service is a no-op.
cd /opt/g8-gw
E=http://etcd-0:2379,http://etcd-1:2379,http://etcd-2:2379
etcdq() { sudo docker run --rm --network g8-gw --user 1000:1000 \
harbor.g8.dev/infra/etcd:v3.5.17 etcdctl --endpoints=$E "$@"; }
sudo docker compose start etcd-0 etcd-1 etcd-2
S=$(sudo docker compose ps --services | wc -l)
for i in $(seq 1 25); do
[ "$(sudo docker compose ps --format '{{.Status}}' | grep -c '(healthy)')" = "$S" ] && break
sleep 6
done
echo "healthy=$(sudo docker compose ps --format '{{.Status}}' | grep -c '(healthy)')/$S unhealthy=$(sudo docker compose ps --format '{{.Status}}' | grep -c unhealthy)"
echo "healthy_endpoints=$(etcdq endpoint health -w simple 2>&1 | grep -c 'is healthy')"
curl -s -o /dev/null --max-time 30 -w "PUT=%{http_code}\n" -K /tmp/g8adm.conf \
-X PUT http://127.0.0.1:9180/apisix/admin/routes/g8-quorum2 \
-H "Content-Type: application/json" \
-d '{"uri":"/quorum2","plugins":{"mocking":{"content_type":"text/plain","response_status":200,"response_example":"RECOVERED"}}}'
sleep 3
echo "9082=$(curl -s -o /dev/null -w '%{http_code}' http://127.0.0.1:9082/quorum2) $(curl -s http://127.0.0.1:9082/quorum2)"
healthy=8/8 unhealthy=0
healthy_endpoints=3
PUT=201
9082=200 RECOVERED
The exam routes are removed:
for r in g8-quorum g8-quorum2; do
curl -s -o /dev/null -w "DELETE $r=%{http_code}\n" -K /tmp/g8adm.conf \
-X DELETE http://127.0.0.1:9180/apisix/admin/routes/$r
done
DELETE g8-quorum=200
DELETE g8-quorum2=200
The admin key file written during verification is removed in a separate step.
Skip it and the key stays in /tmp in cleartext:
rm -f /tmp/g8adm.conf /tmp/g8raw.conf
ls /tmp/g8*.conf 2>/dev/null | wc -l
0
12.6 Restart resilience
cd /opt/g8-gw
sudo docker compose restart
S=$(sudo docker compose ps --services | wc -l)
for i in $(seq 1 25); do
[ "$(sudo docker compose ps --format '{{.Status}}' | grep -c '(healthy)')" = "$S" ] && break
sleep 6
done
echo "healthy=$(sudo docker compose ps --format '{{.Status}}' | grep -c '(healthy)')/$(sudo docker compose ps --services | wc -l) unhealthy=$(sudo docker compose ps --format '{{.Status}}' | grep -c unhealthy)"
for p in 9080 9081 9082; do
echo "$p=$(curl -s -o /dev/null -w '%{http_code}' http://127.0.0.1:$p/smoke)"
done
healthy=8/8 unhealthy=0
9080=200
9081=200
9082=200
12.7 Is log rotation lossless
What is measured is that requests arriving after rotation are written to the
new file; the lines left in the old file are counted too. The count targets
only /smoke lines — counting total lines is unstable because of nginx's delayed
flush.
sudo truncate -s 0 /opt/g8-gw/apisix/logs-0/access.log
for i in 1 2 3; do curl -s -o /dev/null http://127.0.0.1:9080/smoke; done
sleep 3
sudo logrotate -f /etc/logrotate.d/g8-apisix
sleep 3
for i in 1 2 3 4 5; do curl -s -o /dev/null http://127.0.0.1:9080/smoke; done
sleep 3
echo "new=$(sudo grep -c '"uri":"/smoke"' /opt/g8-gw/apisix/logs-0/access.log)"
echo "old=$(sudo grep -c '"uri":"/smoke"' /opt/g8-gw/apisix/logs-0/access.log.1)"
echo "healthy=$(cd /opt/g8-gw && sudo docker compose ps --format '{{.Status}}' | grep -c '(healthy)')/$(cd /opt/g8-gw && sudo docker compose ps --services | wc -l) unhealthy=$(cd /opt/g8-gw && sudo docker compose ps --format '{{.Status}}' | grep -c unhealthy)"
new=5
old=3
healthy=8/8 unhealthy=0
13. Observability
This environment gets its own Elasticsearch and Prometheus: request records go to Elasticsearch, request metrics to Prometheus. There is no shared stack — each environment holds its own data.
| component | version | source |
|---|---|---|
| Elasticsearch | 9.5.2 | harbor.g8.dev/infra/elasticsearch:9.5.2 |
| Prometheus | 3.15.0 | harbor.g8.dev/infra/prometheus:v3.15.0 |
Prometheus starts with --enable-feature=promql-extended-range-selectors:
Analytics counts each bucket exactly with the anchored range selector, which
Prometheus 3.15 takes only behind that flag. From Prometheus 3.16 the selector
is on by default and the line can go; without it, Analytics shows its counts as
estimates.
The added load was measured: together the two services take roughly 2 GB of RAM. On an 8 GB machine they sit comfortably beside APISIX and etcd; the measurement left 5.3 GB free.
This document installs both on one machine. etcd is sensitive to fsync latency and Elasticsearch is a heavy writer. In an installation that needs low latency, move Elasticsearch to a separate disk or a separate machine.
13.1 Services
Two services are added to the services: block of docker-compose.yml:
elasticsearch:
image: harbor.g8.dev/infra/elasticsearch:9.5.2
container_name: g8-elasticsearch
restart: unless-stopped
environment:
- node.name=es-prod
- cluster.name=g8-prod
- discovery.type=single-node
- bootstrap.memory_lock=true
- ES_JAVA_OPTS=-Xms1g -Xmx1g
- xpack.security.enabled=true
- xpack.security.enrollment.enabled=false
- xpack.security.http.ssl.enabled=false
- xpack.security.transport.ssl.enabled=false
- ELASTIC_PASSWORD=${G8_ES_PASSWORD}
- ingest.geoip.downloader.enabled=false
volumes:
- /opt/g8-gw/es-data:/usr/share/elasticsearch/data
ports:
- "9200:9200"
ulimits:
memlock:
soft: -1
hard: -1
nofile:
soft: 65536
hard: 65536
healthcheck:
test: ["CMD-SHELL", "curl -sf -u elastic:$${ELASTIC_PASSWORD} http://127.0.0.1:9200/_cluster/health | grep -qE '\"status\":\"(green|yellow)\"'"]
interval: 15s
timeout: 10s
retries: 10
start_period: 90s
logging: *logging
networks: [g8]
prometheus:
image: harbor.g8.dev/infra/prometheus:v3.15.0
container_name: g8-prometheus
restart: unless-stopped
command:
- --config.file=/etc/prometheus/prometheus.yml
- --storage.tsdb.path=/prometheus
- --storage.tsdb.retention.time=40d
- --storage.tsdb.retention.size=5GB
- --enable-feature=promql-extended-range-selectors
volumes:
- /opt/g8-gw/prometheus/prometheus.yml:/etc/prometheus/prometheus.yml:ro
- /opt/g8-gw/prom-data:/prometheus
ports:
- "9090:9090"
healthcheck:
test: ["CMD", "wget", "-qO-", "http://127.0.0.1:9090/-/healthy"]
interval: 15s
timeout: 5s
retries: 5
start_period: 20s
logging: *logging
networks: [g8]
Data directories and the Elasticsearch password. The password is generated on the machine and written to neither this document nor the repository. It is the store administrator's password and no rule of the gateway carries it: the gateway writes with an account of its own (13.4).
sudo install -d -o 1000 -g 1000 -m 0755 /opt/g8-gw/es-data
sudo install -d -o 65534 -g 65534 -m 0755 /opt/g8-gw/prom-data
sudo install -d -m 0755 /opt/g8-gw/prometheus
printf 'G8_ES_PASSWORD=%s\n' "$(openssl rand -base64 24 | tr -d /+= | head -c 24)" \
| sudo tee -a /opt/g8-gw/.env >/dev/null
sudo chmod 600 /opt/g8-gw/.env
The Prometheus configuration. external_labels marks the environment; when
metrics from several environments meet, this is the only thing that tells them
apart:
global:
scrape_interval: 15s
evaluation_interval: 15s
external_labels:
g8_environment: prod
scrape_configs:
- job_name: apisix
metrics_path: /apisix/prometheus/metrics
static_configs:
- targets: ["apisix-0:9091", "apisix-1:9091", "apisix-2:9091"]
labels:
g8_environment: prod
- job_name: prometheus
static_configs:
- targets: ["127.0.0.1:9090"]
13.2 Start
cd /opt/g8-gw
sudo docker compose up -d
sudo docker ps --format '{{.Names}} {{.Status}}'
g8-apisix-0 Up 2 minutes (healthy)
g8-prometheus Up 2 minutes (healthy)
g8-elasticsearch Up 2 minutes (healthy)
g8-etcd-0 Up 2 days (healthy)
Elasticsearch takes 60-90 seconds to report healthy; that is what
start_period: 90s is for.
13.3 Wiring APISIX
The metrics are enabled through a global rule, which leaves prefer_name off:
route and service on the metrics carry the ids Analytics counts by, and
metrics.*.extra_labels in config.yaml put each route's and service's name
beside them as route_name and service_name. The request records are not wired
here: g8 writes their rule itself when traffic logging is turned on
(13.5).
sudo tee /opt/g8-gw/wire-observability.sh >/dev/null <<'SCRIPT'
#!/bin/bash
set -eu
KEY=$(sudo grep '^G8_APISIX_ADMIN_KEY=' /opt/g8-gw/.env | cut -d= -f2-)
code=$(curl -s -o /dev/null -w '%{http_code}' -H "X-API-KEY: $KEY" \
-X PUT http://127.0.0.1:9180/apisix/admin/global_rules/1 \
-H 'Content-Type: application/json' -d '{"plugins":{"prometheus":{"prefer_name":false}}}')
echo "global_rule -> $code"
SCRIPT
sudo chmod 700 /opt/g8-gw/wire-observability.sh
sudo /opt/g8-gw/wire-observability.sh
global_rule -> 201
prometheus plugin must be enabled; plugin_attr is not enoughplugin_attr.prometheus in config.yaml only sets the address the metrics are
published on. Unless the plugin is enabled on a route or on a global rule, no
request metrics are collected at all. The symptom is treacherous: the Prometheus
target reads up, /apisix/prometheus/metrics returns 200, and yet a query for
apisix_http_status comes back with zero series. That is the measured
behaviour.
13.4 Accounts, the ingest pipeline and the index template in the log store
The commands of this section were run on a development stack on 2026-10-02 (APISIX 3.19.0, Elasticsearch 8.15.3) with the outputs shown. On Elasticsearch 9.5.2 the privileges were measured with roles that name the days by a wildcard, and without the line that reads the template; the roles below name the days by a regular expression, which was measured on 8.15.3 alone. The commands have not been run on this machine yet.
The ingest pipeline, the line of the server's role that reads it and the move of 17.2 were run on 2026-10-04 against a development stack's Elasticsearch 8.15.3 with the outputs shown; that a batch sent twice is stored once was also measured in the lab on Elasticsearch 9.5.2 and OpenSearch 3.9.0.
Two accounts are opened in Elasticsearch, and neither of them is elastic:
| account | used by | may |
|---|---|---|
g8-prod-gateway | the gateway, to deliver the request records | write records into the daily indices of this environment; it reads no record, deletes none and sees nothing of the cluster |
g8-prod-server | the g8 server, to show the records and to clean them up | read the state of the cluster and g8's ingest pipeline, read the daily indices of this environment and delete a day by its name |
The daily indices carry the slug of the environment in g8
(g8-<slug>-access-<yyyy.mm.dd>), so ENV below must be the slug this gateway is
registered under in the g8 console. The roles name the days by a regular
expression and not by g8-prod-access-*: a wildcard would also reach the days of
an environment whose slug begins with prod-access.
sudo tee /opt/g8-gw/store-accounts.sh >/dev/null <<'SCRIPT'
#!/bin/bash
set -eu
ENV=prod
ES=http://127.0.0.1:9200
for who in GATEWAY SERVER; do
grep -q "^G8_ES_${who}_PASSWORD=" /opt/g8-gw/.env || \
printf 'G8_ES_%s_PASSWORD=%s\n' "$who" "$(openssl rand -base64 24 | tr -d /+= | head -c 24)" >> /opt/g8-gw/.env
done
ESP=$(grep '^G8_ES_PASSWORD=' /opt/g8-gw/.env | cut -d= -f2-)
GWP=$(grep '^G8_ES_GATEWAY_PASSWORD=' /opt/g8-gw/.env | cut -d= -f2-)
SVP=$(grep '^G8_ES_SERVER_PASSWORD=' /opt/g8-gw/.env | cut -d= -f2-)
put() {
code=$(curl -s -o /dev/null -w '%{http_code}' -u "elastic:$ESP" -X PUT "$ES$1" \
-H 'Content-Type: application/json' --data-binary @-)
echo "$1 -> $code"
}
DAYS="/g8-$ENV-access-[0-9]{4}[.][0-9]{2}[.][0-9]{2}/"
put /_security/role/g8-$ENV-gateway <<EOF
{"indices":[{"names":["$DAYS"],"privileges":["index","auto_configure"]}]}
EOF
put /_security/role/g8-$ENV-server <<EOF
{"cluster":["monitor","read_pipeline"],"indices":[{"names":["$DAYS"],
"privileges":["read","view_index_metadata","monitor","delete_index"]}]}
EOF
put /_security/user/g8-$ENV-gateway <<EOF
{"password":"$GWP","roles":["g8-$ENV-gateway"]}
EOF
put /_security/user/g8-$ENV-server <<EOF
{"password":"$SVP","roles":["g8-$ENV-server"]}
EOF
SCRIPT
sudo chmod 700 /opt/g8-gw/store-accounts.sh
sudo /opt/g8-gw/store-accounts.sh
/_security/role/g8-prod-gateway -> 200
/_security/role/g8-prod-server -> 200
/_security/user/g8-prod-gateway -> 200
/_security/user/g8-prod-server -> 200
The script can be run again: it keeps the passwords it finds in /opt/g8-gw/.env.
The password of g8-prod-gateway is part of the gateway's configuration: whoever
can read the global rules of the gateway can obtain it. That is why this account
can only add records. When traffic logging is turned on with a plain password, g8
tries the account and refuses one that can read the cluster or a record; a
password given as a secret reference is resolved on the gateway and is not tried.
create_doc is not enough for the account either: the plugin sends an index
action, and with create_doc every record is refused while the request itself
answers 200.
The ingest pipeline stores each record under its request id. The gateway's
logger sends the records in batches and sends a batch again when the store
answers it late, and by then the store has usually written the first one: under
the request id, the second copy replaces the first instead of adding a record.
The index template below sends every record through this pipeline, so the
pipeline is applied first. A store that holds the template and not the pipeline
refuses every record while the gateway's bulk request still answers 200; g8
checks that the store holds this version of the pipeline and does not turn
traffic logging on without it. The store needs at least one node with the
ingest role, which a node carries by default.
ESP=$(sudo grep '^G8_ES_PASSWORD=' /opt/g8-gw/.env | cut -d= -f2-)
curl -s -u "elastic:$ESP" -X PUT http://127.0.0.1:9200/_ingest/pipeline/g8-access \
-H 'Content-Type: application/json' --data-binary @- <<'EOF'
{
"description": "g8 request records: a record is keyed by its request id, so a delivery the gateway sends twice is stored once",
"version": 1,
"processors": [
{
"set": {
"field": "_id",
"value": "{{{request_id}}}",
"if": "ctx.request_id instanceof String && ctx.request_id.length() == 32"
}
}
]
}
EOF
{"acknowledged":true}
The index template fixes the type of every field of a request record before the first index of a day is born. The store's administrator applies it once, with the command below. g8 checks that the store holds this version of it and does not turn traffic logging on without it.
ESP=$(sudo grep '^G8_ES_PASSWORD=' /opt/g8-gw/.env | cut -d= -f2-)
curl -s -u "elastic:$ESP" -X PUT http://127.0.0.1:9200/_index_template/g8-access \
-H 'Content-Type: application/json' --data-binary @- <<'EOF'
{
"index_patterns": [
"g8-*-access-*"
],
"priority": 200,
"template": {
"settings": {
"index": {
"number_of_shards": 1,
"codec": "best_compression",
"auto_expand_replicas": "0-1",
"default_pipeline": "g8-access",
"analysis": {
"analyzer": {
"g8_nodes": {
"type": "pattern",
"pattern": "(\\s*,\\s*)|(\\s+:\\s+)",
"lowercase": false
}
}
}
}
},
"mappings": {
"dynamic": false,
"properties": {
"start_time": {
"type": "date",
"format": "epoch_millis"
},
"latency": {
"type": "float"
},
"upstream_latency": {
"type": "float"
},
"apisix_latency": {
"type": "float"
},
"client_ip": {
"type": "ip",
"ignore_malformed": true
},
"route_id": {
"type": "keyword"
},
"service_id": {
"type": "keyword"
},
"upstream": {
"type": "keyword",
"ignore_above": 1024,
"fields": {
"node": {
"type": "text",
"analyzer": "g8_nodes",
"norms": false,
"index_options": "docs"
}
}
},
"consumer": {
"properties": {
"username": {
"type": "keyword"
}
}
},
"request": {
"properties": {
"method": {
"type": "keyword"
},
"uri": {
"type": "wildcard"
},
"url": {
"type": "keyword",
"index": false,
"doc_values": false
},
"size": {
"type": "long"
},
"headers": {
"type": "object",
"enabled": false
},
"querystring": {
"type": "object",
"enabled": false
}
}
},
"request_body": {
"type": "text",
"index": false
},
"response": {
"properties": {
"status": {
"type": "short"
},
"size": {
"type": "long"
},
"headers": {
"type": "object",
"enabled": false
},
"body": {
"type": "text",
"index": false
}
}
},
"server": {
"properties": {
"hostname": {
"type": "keyword"
},
"version": {
"type": "keyword"
}
}
},
"request_id": {
"type": "keyword"
},
"route_name": {
"type": "keyword"
},
"service_name": {
"type": "keyword"
},
"upstream_status": {
"type": "keyword"
},
"x_request_id": {
"type": "keyword"
},
"logged_at": {
"type": "date",
"format": "epoch_second"
}
},
"_meta": {
"g8_records": 2
},
"_source": {
"excludes": [
"response.headers.set-cookie"
]
}
}
}
}
EOF
{"acknowledged":true}
What the two accounts may and may not do, asked of the store itself. The day the check writes is deleted by its last line:
GWP=$(sudo grep '^G8_ES_GATEWAY_PASSWORD=' /opt/g8-gw/.env | cut -d= -f2-)
SVP=$(sudo grep '^G8_ES_SERVER_PASSWORD=' /opt/g8-gw/.env | cut -d= -f2-)
curl -s -o /dev/null -w 'g8-prod-gateway reads the cluster -> %{http_code}\n' -u "g8-prod-gateway:$GWP" http://127.0.0.1:9200/
curl -s -o /dev/null -w 'g8-prod-gateway reads records -> %{http_code}\n' -u "g8-prod-gateway:$GWP" 'http://127.0.0.1:9200/g8-prod-access-*/_search'
curl -s -o /dev/null -w 'g8-prod-gateway writes a day -> %{http_code}\n' -u "g8-prod-gateway:$GWP" -X POST http://127.0.0.1:9200/g8-prod-access-2000.01.01/_doc -H 'Content-Type: application/json' -d '{}'
curl -s -o /dev/null -w 'g8-prod-gateway writes beside its days -> %{http_code}\n' -u "g8-prod-gateway:$GWP" -X POST http://127.0.0.1:9200/g8-prod-access-x-access-2000.01.01/_doc -H 'Content-Type: application/json' -d '{}'
curl -s -o /dev/null -w 'g8-prod-server reads the template -> %{http_code}\n' -u "g8-prod-server:$SVP" http://127.0.0.1:9200/_index_template/g8-access
curl -s -o /dev/null -w 'g8-prod-server reads the pipeline -> %{http_code}\n' -u "g8-prod-server:$SVP" http://127.0.0.1:9200/_ingest/pipeline/g8-access
curl -s -o /dev/null -w 'g8-prod-server reads beside its days -> %{http_code}\n' -u "g8-prod-server:$SVP" 'http://127.0.0.1:9200/g8-prod-access-x-access-2000.01.01/_search'
curl -s -o /dev/null -w 'g8-prod-server writes a record -> %{http_code}\n' -u "g8-prod-server:$SVP" -X POST http://127.0.0.1:9200/g8-prod-access-2000.01.01/_doc -H 'Content-Type: application/json' -d '{}'
curl -s -o /dev/null -w 'g8-prod-server deletes the day -> %{http_code}\n' -u "g8-prod-server:$SVP" -X DELETE http://127.0.0.1:9200/g8-prod-access-2000.01.01
g8-prod-gateway reads the cluster -> 403
g8-prod-gateway reads records -> 403
g8-prod-gateway writes a day -> 201
g8-prod-gateway writes beside its days -> 403
g8-prod-server reads the template -> 200
g8-prod-server reads the pipeline -> 200
g8-prod-server reads beside its days -> 403
g8-prod-server writes a record -> 403
g8-prod-server deletes the day -> 200
elasticsearch-logger writes an index action in its bulk request; data streams
require create and reject index. Hence one dated index per day.
13.5 Turning on the request records
These steps were walked on a development stack on 2026-10-02, with the two accounts of 13.4. They have not been walked against this machine yet.
The request records are turned on from the g8 console, once this gateway is
registered there as the environment prod. No command on this machine writes
the logger's rule.
- Environments › prod › Edit › Observability. Elasticsearch URL: the address
the g8 server reaches this store at.
Username
g8-prod-server, password the value ofG8_ES_SERVER_PASSWORDin/opt/g8-gw/.env. Test connection, then save. - Observe › Traffic Explorer › Settings. The Status card must read Log store Connected, Index template Applied and Ingest pipeline Applied.
- Turn on traffic logging. Log store address, as the gateway reaches it:
http://elasticsearch:9200. Usernameg8-prod-gateway, password the value ofG8_ES_GATEWAY_PASSWORDin/opt/g8-gw/.env. Turn on.
g8 writes one global rule, g8-request-records, beside rule 1: the logger with
the daily index of this environment, the fields the records need and the default
masks. What is recorded and masked is changed afterwards on the same Settings
page. Where the gateway writes is shown on the Status card as Gateway writes
to; Change address gives the logger another address and account for the
same store, through the same checks of the account as turning on.
13.6 Retention
A log store does not shrink on its own. With no limit the disk fills eventually, and past 95% Elasticsearch turns its indices read-only.
g8 deletes the request records: once a day it deletes, by its name, every daily index older than the number of days the environment keeps — 30 unless it is changed under Traffic Explorer › Settings › Retention › Change — and writes each deletion to the audit log. The store carries no lifecycle policy for these indices; a second deleter would delete days g8 still shows.
A day is named by its UTC date, by the gateway that writes it and by g8 that reads and deletes it.
On a single-node Elasticsearch a replica can never be allocated. The template
sets auto_expand_replicas: 0-1: an index has no replica on one node and one as
soon as a second node joins. The verification below must show 0 and green.
Prometheus is given two limits at once: --storage.tsdb.retention.time=40d and
--storage.tsdb.retention.size=5GB. Whichever fills first does the cutting; a
time limit alone does not keep the disk from filling. Forty days keep the whole
30-day window of Analytics.
13.7 Verification
The output below was recorded on this machine while the rule of the earlier guide
was in place; the index it shows is named by the machine's local day. With the
rule g8 writes, the commands were run on a development stack on 2026-10-02: the
index of the UTC day showed the requests sent, 0 and green. They have not
been run on this machine since.
ESP=$(sudo grep '^G8_ES_PASSWORD=' /opt/g8-gw/.env | cut -d= -f2-)
for p in 9080 9081 9082; do
for i in $(seq 1 10); do curl -s -o /dev/null http://127.0.0.1:$p/smoke; done
done
sleep 20
curl -s -u "elastic:$ESP" 'http://127.0.0.1:9200/_cat/indices/g8-*?v&h=index,docs.count,rep,health'
curl -s 'http://127.0.0.1:9090/api/v1/query?query=count+by+(instance)+(apisix_http_status)' \
| grep -o '"instance":"[^"]*"' | sort
index docs.count rep health
g8-prod-access-2026.08.29 25 0 green
The index must show 0 in the rep column and green in health, and the
last query must print three distinct instance values.
There is no load balancer in front of the APISIX nodes in this install: 9080
reaches only apisix-0, 9081 apisix-1, 9082 apisix-2. Send requests to one
port and only one node produces metrics; the plugin on the other two may never
have run and you would not see it. If count by (instance) does not show all
three, the global rule has not reached that node.
Traffic Explorer in the g8 console must list the same requests.
Analytics reads two things from these metrics: each route under its id with its name beside it, and each bucket counted exactly with the anchored range selector:
curl -sG http://127.0.0.1:9090/api/v1/query --data-urlencode 'query=count by (route, route_name) (apisix_http_status)'
curl -sG http://127.0.0.1:9090/api/v1/query --data-urlencode 'query=sum(increase(apisix_http_status[5m] anchored))'
The first query lists every route that answered with its id in route and its
name in route_name; a route that carries a name means a copy of the
prometheus plugin has prefer_name on, and Analytics reads that route empty.
The second must answer "status":"success"; a Prometheus started without
--enable-feature=promql-extended-range-selectors answers anchored modifier is experimental and not enabled, and Analytics marks its counts as estimates. Both
queries were run on this machine on 2026-10-04 against Prometheus 3.15.0: every
route with a name showed it in route_name, and the anchored query answered
success.
The gateway sends a batch of records once more after the first failure
(max_retry_count 1 in the rule g8 writes) and then drops it.
The loss was measured with the rule of the earlier guide, which tried three
times: on a production installation during a password change,
20 requests that arrived while Elasticsearch was restarting appeared in the
metrics but never reached the index.
Read the loss from APISIX's error log, not from a counter difference:
grep -c 'elasticsearch-logger' <error.log>
Subtracting the document count from the metric counter is not reliable and produces false alarms. A measured example: counter 88, documents 43, difference 45 — with not one delivery error in the log. The difference has three causes and none of them is loss: the counter counts from the moment APISIX started (requests from before the plugin was enabled are in the counter but not the index), a restart resets the counter while the index persists, and the two sides have different retention. The difference is a hint only within a single window — no APISIX restart, nothing deleted by retention, both plugins enabled together.
An installation that cannot afford losing records must put a buffer in between
(kafka-logger, or a local file-logger plus a separate shipper).
Restarts were measured to lose nothing: across a container restart and a full
machine reboot the document count did not change, and Prometheus data from before
the restart was still queryable. Persistence comes from the es-data and
prom-data bind mounts; delete those two directories and the data is gone.
14. Diagnostics
logs output is cumulative: it holds everything since the container or pod
was born. APISIX prints connection refused while waiting for etcd at startup,
and the same lines land again for members stopped during the quorum drill. Those
lines describe the past, not the present state.
Measured on a healthy installation in this environment:
request etcd endpoint 'http://etcd-1:2379/version' error, connection refused
error.log lines: 0
An empty error.log is normal too — APISIX's own error log is a separate file,
and the startup messages go to stdout rather than there.
The present state is what the verification section's counts tell you; logs is
for diagnosis, not proof of health. Decide whether a line is current from its
timestamp.
cd /opt/g8-gw
sudo docker compose ps
sudo docker compose logs --tail 50 apisix-0
sudo docker compose logs --tail 50 etcd-0
sudo tail -n 50 /opt/g8-gw/apisix/logs-0/error.log
The cluster tables meant for human reading — member IDs and timings differ on every installation, which is why these outputs are not given as fixed expected output:
E=http://etcd-0:2379,http://etcd-1:2379,http://etcd-2:2379
sudo docker exec g8-etcd-0 etcdctl --endpoints=$E member list -w table
sudo docker exec g8-etcd-0 etcdctl --endpoints=$E endpoint status -w table
sudo docker exec g8-etcd-0 etcdctl --endpoints=$E endpoint health -w table
Prometheus is tested separately on all three nodes:
for p in 9091 9092 9093; do
echo "$p=$(curl -s -o /dev/null -w '%{http_code}' http://127.0.0.1:$p/apisix/prometheus/metrics) $(curl -s http://127.0.0.1:$p/apisix/prometheus/metrics | grep -c '^apisix_')"
done
9091=200 87
9092=200 87
9093=200 87
15. Backup and restore
The only state to back up is in etcd; APISIX holds none. On the file side, what
needs backing up is config.yaml, docker-compose.yml and .env.
The backup is taken by a one-off container without touching the running
services. snapshot save requires a single endpoint
(18.1) — but that endpoint is not hardcoded, it is
picked from the running ones. The backup_from line varies per run.
cd /opt/g8-gw
E1=$(sudo docker compose ps --services --filter status=running | grep '^etcd-' | head -1)
echo "backup_from=$E1"
sudo docker run --rm --network g8-gw --user 1000:1000 \
-v /opt/g8-gw/backups:/backups harbor.g8.dev/infra/etcd:v3.5.17 \
etcdctl --endpoints="http://$E1:2379" snapshot save /backups/etcd-manual.db
sudo docker run --rm --user 1000:1000 -v /opt/g8-gw/backups:/backups \
harbor.g8.dev/infra/etcd:v3.5.17 etcdutl snapshot status /backups/etcd-manual.db -w table
Daily automated backup. The backup is not tied to a fixed member: it looks
for the first healthy one and, if none is healthy, exits 1 without producing a
file. Had etcd-0 been hardcoded, backups would stop silently on the day that
member goes down and nobody would notice (18.7).
sudo tee /opt/g8-gw/etcd-backup.sh >/dev/null <<'SCRIPT'
#!/bin/sh
set -eu
IMAGE=harbor.g8.dev/infra/etcd:v3.5.17
NET=g8-gw
DIR=/opt/g8-gw/backups
STAMP=$(date +%Y%m%d-%H%M)
ls -1t "$DIR"/etcd-*.db 2>/dev/null | tail -n +15 | xargs -r rm -f
for e in etcd-0 etcd-1 etcd-2; do
if docker run --rm --network "$NET" --user 1000:1000 "$IMAGE" \
etcdctl --endpoints="http://$e:2379" --command-timeout=30s endpoint health >/dev/null 2>&1; then
docker run --rm --network "$NET" --user 1000:1000 -v "$DIR":/backups "$IMAGE" \
etcdctl --endpoints="http://$e:2379" --command-timeout=120s snapshot save "/backups/etcd-$STAMP.db"
echo "snapshot taken from $e"
exit 0
fi
done
echo "no healthy etcd endpoint reachable; NO backup was taken" >&2
exit 1
SCRIPT
sudo chmod 0755 /opt/g8-gw/etcd-backup.sh
sudo tee /etc/systemd/system/g8-etcd-backup.service >/dev/null <<'EOF'
[Unit]
Description=g8 etcd snapshot backup
After=docker.service
Requires=docker.service
[Service]
Type=oneshot
TimeoutStartSec=300
ExecStart=/opt/g8-gw/etcd-backup.sh
EOF
sudo tee /etc/systemd/system/g8-etcd-backup.timer >/dev/null <<'EOF'
[Unit]
Description=g8 etcd snapshot backup (daily)
[Timer]
OnCalendar=daily
Persistent=true
[Install]
WantedBy=timers.target
EOF
sudo systemctl daemon-reload
sudo systemctl enable --now g8-etcd-backup.timer
That the timer is installed does not prove a backup was taken. Run the service
once by hand and read which member it came from; the member name varies per
run, success does not:
sudo systemctl start g8-etcd-backup.service
systemctl show -p Result --value g8-etcd-backup.service
echo "fresh_backup=$(find /opt/g8-gw/backups -name 'etcd-*.db' -mmin -2 | wc -l)"
success
fresh_backup=1
15.1 Restore
Restoring a three-member cluster from a snapshot means replacing all three data directories at once. The cluster is stopped and the old data directories are moved aside, not deleted — if the restore does not go as expected, that is the only place to go back to.
The --data-dir path needs care: /opt/g8-gw is mounted into the container as
/target and the data is written into /target/etcd-data-$n. Writing one
directory deeper (like /restore/data) does not match etcd's
ETCD_DATA_DIR: /etcd-data expectation, and the cluster comes up with no data.
cd /opt/g8-gw
sudo docker compose down
for n in 0 1 2; do sudo mv etcd-data-$n etcd-data-$n.bozuk; done
for n in 0 1 2; do
sudo docker run --rm --user 1000:1000 \
-v /opt/g8-gw/backups:/backups -v /opt/g8-gw:/target \
harbor.g8.dev/infra/etcd:v3.5.17 \
etcdutl snapshot restore /backups/etcd-manual.db \
--name etcd-$n \
--initial-cluster etcd-0=http://etcd-0:2380,etcd-1=http://etcd-1:2380,etcd-2=http://etcd-2:2380 \
--initial-advertise-peer-urls http://etcd-$n:2380 \
--data-dir /target/etcd-data-$n
done
sudo docker compose up -d
That the restore actually restored is verified separately. A cluster being up is not sufficient evidence: a wrong restore also produces a healthy cluster, just an empty one.
cd /opt/g8-gw
E=http://etcd-0:2379,http://etcd-1:2379,http://etcd-2:2379
etcdq() { sudo docker run --rm --network g8-gw --user 1000:1000 \
harbor.g8.dev/infra/etcd:v3.5.17 etcdctl --endpoints=$E "$@"; }
echo "healthy=$(sudo docker compose ps --format '{{.Status}}' | grep -c '(healthy)')/$S unhealthy=$(sudo docker compose ps --format '{{.Status}}' | grep -c unhealthy)"
echo "healthy_endpoints=$(etcdq endpoint health -w simple 2>&1 | grep -c 'is healthy')"
curl -s http://127.0.0.1:9080/smoke -w "\nstatus=%{http_code}\n"
healthy=8/8 unhealthy=0
healthy_endpoints=3
G8-GW-PROD-OK
status=200
To prove the data returned to the right moment, write one marker before and one after the backup; after the restore the first must be there and the second must be gone. Measured in this environment:
marker-A=before-backup
marker-B=
marker-B coming back empty is the result you are looking for — it shows that
everything written after the snapshot is gone. If both come back, no restore
happened and the cluster came up on the old data.
15.2 Defragmentation
Compaction removes old revisions logically but does not shrink the database
file — the freed pages stay inside etcd and never return to the filesystem.
Since the quota watches dbSize, the file keeps growing even when compaction is
working perfectly, and the quota fills sooner or later. The extreme measured:
dbSizeInUse at 41 kB while dbSize sat at 98 MB.
Defragmentation therefore has to be scheduled. Weekly is enough.
sudo tee /opt/g8-gw/etcd-defrag.sh >/dev/null <<'SCRIPT'
#!/bin/sh
set -eu
IMAGE=harbor.g8.dev/infra/etcd:v3.5.17
NET=g8-gw
for e in etcd-0 etcd-1 etcd-2; do
if docker run --rm --network "$NET" --user 1000:1000 "$IMAGE" \
etcdctl --endpoints="http://$e:2379" --command-timeout=30s endpoint health >/dev/null 2>&1; then
echo "defrag $e"
docker run --rm --network "$NET" --user 1000:1000 "$IMAGE" \
etcdctl --endpoints="http://$e:2379" --command-timeout=900s defrag
sleep 15
else
echo "skip $e (unhealthy)"
fi
done
docker run --rm --network "$NET" --user 1000:1000 "$IMAGE" \
etcdctl --endpoints=http://etcd-0:2379,http://etcd-1:2379,http://etcd-2:2379 \
--command-timeout=30s alarm list
SCRIPT
sudo chmod 0755 /opt/g8-gw/etcd-defrag.sh
sudo tee /etc/systemd/system/g8-etcd-defrag.service >/dev/null <<'EOF'
[Unit]
Description=g8 etcd defragmentation
After=docker.service
Requires=docker.service
[Service]
Type=oneshot
TimeoutStartSec=1800
ExecStart=/opt/g8-gw/etcd-defrag.sh
EOF
sudo tee /etc/systemd/system/g8-etcd-defrag.timer >/dev/null <<'EOF'
[Unit]
Description=g8 etcd defragmentation (weekly)
[Timer]
OnCalendar=Sun 03:00
Persistent=true
[Install]
WantedBy=timers.target
EOF
sudo systemctl daemon-reload
sudo systemctl enable --now g8-etcd-defrag.timer
Verification:
sudo systemctl start g8-etcd-defrag.service
systemctl show -p Result --value g8-etcd-defrag.service
success
Defrag blocks that member briefly, which is why the members are processed one at a time. Defragmenting all three at once drops quorum.
Defrag temporarily needs free space equal to the database size; that is one reason the quota must be meaningfully smaller than the disk.
15.3 Getting backups off the node
Until now the backups sit on the same disk as the data they protect. If the disk goes, both go with it — the backup does not protect against the one event it promises to.
Direction matters: the remote machine must do the copying. Had the gateway pushed to the remote machine, the gateway would need write access there; when the gateway is compromised, the backups go too. Pulling means the gateway holds no secret at all.
On the gateway, a restricted command is defined that grants no shell and only produces the backup stream:
printf '#!/bin/sh\nexec /usr/bin/tar -C /opt/g8-gw/backups -cf - .\n' \
| sudo tee /usr/local/bin/g8-backup-stream >/dev/null
sudo chmod 0755 /usr/local/bin/g8-backup-stream
echo 'cosmonot ALL=(root) NOPASSWD: /usr/local/bin/g8-backup-stream' \
| sudo tee /etc/sudoers.d/g8-backup-stream >/dev/null
sudo chmod 0440 /etc/sudoers.d/g8-backup-stream
sudo visudo -c
The remote machine's public key is added to authorized_keys with a command
restriction; a session opened with that key gets no shell, only that command:
mkdir -p ~/.ssh && chmod 700 ~/.ssh
cat >> ~/.ssh/authorized_keys <<'EOF'
command="sudo /usr/local/bin/g8-backup-stream",no-pty,no-port-forwarding,no-agent-forwarding,no-X11-forwarding ssh-ed25519 AAAA... g8-obs-backup-pull
EOF
chmod 600 ~/.ssh/authorized_keys
On the remote machine (backup-host in this lab), the pull script:
sudo tee /usr/local/bin/g8-pull-backups.sh >/dev/null <<'SCRIPT'
#!/bin/sh
set -u
DEST=/var/backups/g8-remote
KEY=/home/cosmonot/.ssh/id_ed25519
RC=0
for spec in test-vm:192.0.2.107 prod:192.0.2.108; do
name=${spec%%:*}
host=${spec##*:}
rm -rf "$DEST/$name.new"
mkdir -p "$DEST/$name.new"
if ssh -i "$KEY" -o BatchMode=yes -o ConnectTimeout=30 \
-o StrictHostKeyChecking=accept-new \
cosmonot@"$host" 2>/dev/null | tar -C "$DEST/$name.new" -xf - 2>/dev/null \
&& [ -n "$(ls -A "$DEST/$name.new")" ]; then
rm -rf "$DEST/$name.prev"
[ -d "$DEST/$name" ] && mv "$DEST/$name" "$DEST/$name.prev"
mv "$DEST/$name.new" "$DEST/$name"
echo "pulled $name: $(ls -1 "$DEST/$name" | wc -l) file(s)"
else
rm -rf "$DEST/$name.new"
echo "FAILED $name" >&2
RC=1
fi
done
exit $RC
SCRIPT
sudo chmod 0755 /usr/local/bin/g8-pull-backups.sh
sudo mkdir -p /var/backups/g8-remote
And the timer:
sudo tee /etc/systemd/system/g8-pull-backups.service >/dev/null <<'EOF'
[Unit]
Description=g8 pull gateway etcd backups
After=network-online.target
Wants=network-online.target
[Service]
Type=oneshot
TimeoutStartSec=900
ExecStart=/usr/local/bin/g8-pull-backups.sh
EOF
sudo tee /etc/systemd/system/g8-pull-backups.timer >/dev/null <<'EOF'
[Unit]
Description=g8 pull gateway etcd backups (daily)
[Timer]
OnCalendar=daily
RandomizedDelaySec=1800
Persistent=true
[Install]
WantedBy=timers.target
EOF
sudo systemctl daemon-reload
sudo systemctl enable --now g8-pull-backups.timer
Verification. "There is a file" is not a backup; ask whether the pulled copy is identical to the source:
sudo systemctl start g8-pull-backups.service
systemctl show -p Result --value g8-pull-backups.service
sudo sha256sum /var/backups/g8-remote/prod/etcd-manual.db | cut -c1-16
success
27e8a4200d411a2d
The same sha256 shows the copy arrived intact. Compare it with the sha256 of the file on the source.
16. Uninstalling
Delete the environment in the g8 console first, or clear its log store there: g8 connects to the store once a day to delete the days past the retention, and records a failed cleanup every day for as long as the environment points at a store that is gone.
cd /opt/g8-gw
sudo docker compose down
sudo systemctl disable --now g8-etcd-backup.timer
sudo rm -f /etc/systemd/system/g8-etcd-backup.service /etc/systemd/system/g8-etcd-backup.timer
sudo systemctl daemon-reload
sudo rm -f /etc/logrotate.d/g8-apisix
Deleting the data and the secrets is a separate decision; the commands above do not touch them:
sudo rm -rf /opt/g8-gw
Verify that the uninstall completed:
sudo docker ps -a --format '{{.Names}}' | grep -c '^g8-' || true
ss -lntH | awk '{print $4}' | grep -cE ':(908[0-2]|918[0-2]|909[1-3]|9200|9300|9090)$' || true
0
0
17. Upgrading
A version upgrade was not attempted in this environment. The sequence below is a design, not evidence; it must first be applied in a test environment.
- Take an etcd backup with
snapshot saveand verify it withsnapshot status. - Pull the new image tag from Harbor.
- Renew the APISIX nodes one at a time:
docker compose up -d --force-recreate apisix-0, wait for it to become healthy, run the 12.3 verification, then move to the next. That way there is no outage. - Upgrading etcd is separate work and is done without skipping versions; members
are renewed one at a time and
endpoint healthis verified to return three at every step.
Rollback path: the old image tags are still in Harbor; the same steps are repeated with the old tag.
On 2026-09-30 the three APISIX nodes moved from 3.18.0 to 3.19.0 in step 3's
order: one node at a time, each healthy and answering Server: APISIX/3.19.0 on
its own port (9080, 9081, 9082) before the next; etcd was not touched. Whether
step 1's backup and the 12.3 verification ran for that
change is not recorded, so the sequence stays unverified.
17.1 Request records of an installation made with the earlier guide
This move was rehearsed on a development stack on 2026-10-02: an installation made as the earlier guide wrote it, then the steps below in this order, with the outputs shown. It has not been carried out on this machine yet.
The earlier version of this guide wrote the logger into global rule 1 by hand:
with the elastic account, an index named g8-prod-access- and the machine's
local day, and a lifecycle policy that deleted the indices. Such an installation
is moved in this order:
Settings offers Turn on traffic logging as soon as the pipeline and the template
are applied.
Turning on over the old rule completes it in place: the account is neither asked
for nor tried, and elastic stays in the rule.
- Open the two accounts (13.4).
- Connect the store in the g8 console with
g8-prod-server(step 1 of 13.5). Settings then reads Index template Outdated, Ingest pipeline Missing and the logger rule Customized. From its next daily cleanup on, g8 deletes the days past the retention. - Apply the ingest pipeline and then the index template (13.4). They apply to the days born after them.
- Take the lifecycle policy off the indices and delete it:
ESP=$(sudo grep '^G8_ES_PASSWORD=' /opt/g8-gw/.env | cut -d= -f2-)
curl -s -u "elastic:$ESP" -X PUT 'http://127.0.0.1:9200/g8-*-access-*/_settings' \
-H 'Content-Type: application/json' -d '{"index.lifecycle.name":null}'
curl -s -u "elastic:$ESP" -X DELETE http://127.0.0.1:9200/_ilm/policy/g8-access-retention
{"acknowledged":true}{"acknowledged":true}
- The policy kept the indices 30 days. g8 keeps them 30 days unless that is changed: set the number of days this store is sized for under Traffic Explorer › Settings › Retention › Change.
- Run the whole block of 13.3 again. It writes
wire-observability.shanew before it runs it: the file on this machine is the earlier guide's and would write the logger back. The block printsglobal_rule -> 200here; rule1then carries the metrics alone and Settings reads the logger rule Off. Only then turn traffic logging on, withg8-prod-gateway(steps 2 and 3 of 13.5). - Change the password of
elastic: it was part of the gateway's configuration for as long as the old rule existed.
No record is written between the two halves of step 6. Every day written before the move, and the day of the move itself, was born without g8's template and without the fields its records carry: Traffic Explorer cannot list those days, and Settings counts them as written under another version. They stay in the store until the retention deletes them. The records are listed from the first day whose index is born after the move.
g8 lists and deletes the days of the environment's own slug alone. If this
gateway is registered in g8 under another slug than prod, the days the old rule
wrote have no deleter once the policy is gone: delete them by their names when
they are no longer needed.
17.2 The ingest pipeline on an installation made before it
This move was rehearsed on 2026-10-04 against a development stack's Elasticsearch 8.15.3: an installation with the earlier template and a day born under it, then the steps below in this order, with the outputs shown. It has not been carried out on this machine yet.
An installation made before this version of the guide holds the index template without the ingest pipeline, and its server account may not read pipelines. Settings then reads Index template Outdated and Ingest pipeline Cannot be read. The gateway keeps writing records meanwhile, copies included, and the settings can still be saved; g8 refuses only to turn logging on anew until both read Applied. The move, in this order:
- Run the whole block that writes the accounts script again
(13.4). It rewrites the saved script with the new
server role and runs it, keeping the passwords; every line answers
200. Running the saved script alone writes the old role again. - Apply the ingest pipeline (13.4).
- Apply the index template (13.4). The days born after it send their records through the pipeline.
- Send the day of today, born under the earlier template, through the pipeline at once. On a store that serves several environments, run it once for each environment's slug:
ESP=$(sudo grep '^G8_ES_PASSWORD=' /opt/g8-gw/.env | cut -d= -f2-)
curl -s -u "elastic:$ESP" -X PUT "http://127.0.0.1:9200/g8-prod-access-$(date -u +%Y.%m.%d)/_settings" \
-H 'Content-Type: application/json' -d '{"index.default_pipeline":"g8-access"}'
{"acknowledged":true}
A store that holds no record of today yet answers 404 index_not_found_exception
here: today's day will be born under the new template, and nothing more is
needed. Settings then reads both Applied. A day written before the move may hold copies
of a batch the gateway delivered twice; it stays as it is until the retention
deletes it.
18. Traps hit during installation
18.1 snapshot save requires a single endpoint
Symptom:
Error: snapshot must be requested to one selected node, not multiple [http://etcd-0:2379 http://etcd-1:2379 http://etcd-2:2379]
Because the cluster's other commands (member list, endpoint health) accept
several endpoints, the same variable gets passed to snapshot save as well. A
snapshot is a copy of one member's state; which member it is taken from must be
chosen explicitly.
18.2 The Admin API does not fail fast on quorum loss
Symptom: the client hangs without receiving a response; an automation client with a 30-second timeout sees a hang, not an error.
PUT=000 sure=40.002921s
The etcd.timeout: 30 in config.yaml applies to a single request. APISIX
tries the three endpoints in turn and retries on each; in this environment the
total was measured across three consecutive requests as:
req1: PUT=503 time=104.881558s
req2: PUT=503 time=86.135962s
req3: PUT=503 time=73.792120s
The last one matters: the requests do not get faster. As long as APISIX can reach the surviving etcd but that etcd returns 503 for lack of quorum, every new request pays the same cost. The body returned varies per attempt:
{"error_msg":"http://etcd-2:2379: no route to host"}
{"error_msg":"invalid response code: 503"}
The behaviour differs in the OKD installation — there only the first request is
slow (04-okd, 14.7). Do not write one rule for both platforms; the measurement
belongs to its environment.
Automation clients' timeouts must be chosen accordingly, or quorum must be
checked with endpoint health before writing.
18.3 The health check fills the access log
Symptom: access.log grows even with no traffic at all; almost every line
carries an empty uri and "status":"400".
sudo grep -c '"status":"400"' /opt/g8-gw/apisix/logs-0/access.log
echo > /dev/tcp/127.0.0.1/9080 opens a TCP connection and sends a newline.
nginx treats that as a malformed request line, returns 400, and logs it. At a
ten-second interval that is 8640 lines per node per day — 25920 across three
nodes.
The control API (9090) does not write to the access log, which is why the health check points there. The measurement and its negative control:
B=$(sudo wc -l < /opt/g8-gw/apisix/logs-0/access.log)
for i in $(seq 1 20); do sudo docker exec g8-apisix-0 bash -c 'echo > /dev/tcp/127.0.0.1/9090'; done
sleep 8
echo "delta_9090=$(( $(sudo wc -l < /opt/g8-gw/apisix/logs-0/access.log) - B ))"
B=$(sudo wc -l < /opt/g8-gw/apisix/logs-0/access.log)
for i in $(seq 1 20); do sudo docker exec g8-apisix-0 bash -c 'echo > /dev/tcp/127.0.0.1/9080'; done
sleep 8
echo "delta_9080=$(( $(sudo wc -l < /opt/g8-gw/apisix/logs-0/access.log) - B ))"
delta_9090=0
delta_9080=20
Do not treat delta_9090=0 as evidence without running the negative control: the
measuring tool may simply not have run. The sleep 8 is required too — nginx
flushes the access log with a delay, and with a two-second wait the count comes
out as 16 instead of 20.
18.4 The Docker log driver does not cover APISIX's logs
Symptom: docker inspect shows max-size: 10m and the host disk fills
anyway. docker logs g8-apisix-0 is nearly empty.
The x-logging anchor rotates only the container's stdout/stderr. APISIX,
however, writes to a file because of the access_log path in config.yaml, and
that file is in the bind-mounted directory. The rate measured in this environment
is roughly 140 bytes per request; at 1000 requests per second that is ~12 GB a
day. The fix is the host logrotate in section 10.
18.5 Stopping a container also removes its DNS name
Symptom: the etcd log shows a name resolution error rather than a network error.
failed to parse domain: etcd-2, error: failed to query the DNS server
docker compose stop also removes the container from the network; its name no
longer resolves. In a real network partition the name resolves but the connection
cannot be made — the error message is different. When drilling member loss, keep
in mind that these two are not the same thing.
18.6 The etcd image contains no shell
Symptom:
OCI runtime exec failed: exec failed: unable to start container process: exec: "sh": executable file not found in $PATH
The etcd image carries only the etcd binaries. docker exec g8-etcd-0 sh -c …
does not work; commands are given directly (docker exec g8-etcd-0 etcdctl …).
Measurements that need a shell inside the container are taken from the host via
/proc/<pid>.
18.7 Frozen values: being the transcript of a single run
Symptom: the document works on one run and not on another — and on the run where it does not work, it does not produce an error, it produces a wrong number.
When this document was first written it was the transcript of one successful run. Three values incidental to that run were frozen in, and all three turned into defects:
| what was frozen | why it is wrong |
|---|---|
logrotate -d … | head -2 | The lines you want sit behind the banner, at a position that varies by machine (16 and 36 measured); head -2 catches the banner |
stop etcd-1 / start etcd-1 etcd-2 in 12.5 | Consistent only when the leader is etcd-2. If the leader is etcd-0 that member is never restarted and the installation ends with zero redundancy; if the leader is etcd-1 the stop is a no-op and the drill proves nothing |
Running cluster queries through docker exec g8-etcd-0 | The node the drill may stop. If the container is dead the exec fails at the daemon, 2>&1 folds the error into the grep, and the result is healthy_endpoints=0 — a dead probe looks like a sick cluster |
The rule common to all three: a command must depend on the content of the output, not its position; and an argument must be computed from the current state. If you must write a fixed name, verify in the same step that the name is still valid.
Measured: with etcd-0 stopped, the two forms say this about the same cluster:
docker exec g8-etcd-0 ... -> 0
tek seferlik konteyner -> 2
18.8 The NOSPACE alarm is a one-way door
Symptom: writes are refused with etcdserver: mvcc: database space exceeded.
Nobody notices, because the data plane keeps returning 200 — and once it is
noticed, freeing space is not enough.
The recovery chain was measured, and all three steps are required:
159 writes -> dbSize=16789504 alarm=NOSPACE
delete + compact -> dbSize=16789504 write refused
defrag -> dbSize=122880 write refused
alarm disarm -> write OK
There are three separate misconceptions here, each verified individually:
- Deleting keys does not free space. etcd keeps the revision history; a delete writes a new revision.
- Compaction does not return disk space.
dbSizeInUsedrops,dbSizedoes not move at all — the freed pages stay free inside etcd and never return to the filesystem. Since the quota watchesdbSize, the quota is still full. The extreme measured in this environment:dbSizeInUseat 41 kB whiledbSizesat at 98 MB. - Defrag returns the space but does not lower the alarm. The alarm is sticky: it survives a process restart, a container recreation and a host reboot.
Recovery:
sudo /opt/g8-gw/etcd-defrag.sh
sudo docker run --rm --network g8-gw --user 1000:1000 harbor.g8.dev/infra/etcd:v3.5.17 \
etcdctl --endpoints=http://etcd-0:2379,http://etcd-1:2379,http://etcd-2:2379 alarm disarm
Leave alarm disarm for last: disarming without actually freeing space puts you
straight back where you were on the next write.
The lasting fix is scheduled defragmentation — see
long-term operation. With no defrag scheduled, dbSize keeps
growing even when compaction is working correctly, and the quota fills sooner or
later.
18.9 A member that loses its disk cannot rejoin on its own
Symptom: a member loses its data directory (disk failure, accidental
deletion, a lost PVC). When it comes back it loops forever, logging
cluster ID mismatch or member has already been bootstrapped.
What makes it insidious: member list still shows that member as started
and the data plane keeps returning 200. The cluster silently drops to 2/3, and
losing one more member takes quorum with it.
The cause is ETCD_INITIAL_CLUSTER_STATE: new hardcoded in the manifest. Seeing
an empty data directory, etcd tries to bootstrap a new cluster rather than
join the existing one.
The recovery chain was measured:
STATE=new + bos veri dizini -> cluster ID mismatch (uye asla katilmaz)
member remove + olcek dusur -> veri dizini gercekten bos
member add (saglikli kume) -> ETCD_INITIAL_CLUSTER=...
STATE=existing, tek acilis -> healthy=3/3
STATE=new'e donus -> healthy=3/3, veri yerinde
The critical detail: STATE is only read while the data directory is empty.
So existing is needed for exactly one start; once the member has joined it
runs indefinitely under its normal definition.
Do not wipe the data directory without stopping the member first: while the container is up, etcd rebuilds the directory immediately.
member add refuses with etcdserver: unhealthy cluster while the remaining
cluster is unhealthy — which is why endpoint health is asked first.
Recovery:
cd /opt/g8-gw
IMG=harbor.g8.dev/infra/etcd:v3.5.17
E=http://etcd-0:2379,http://etcd-1:2379,http://etcd-2:2379
etcdq() { sudo docker run --rm --network g8-gw --user 1000:1000 $IMG etcdctl --endpoints=$E "$@"; }
ID=$(etcdq member list -w simple | grep 'etcd-2' | cut -d, -f1)
sudo docker compose stop etcd-2
etcdq member remove "$ID"
sudo rm -rf /opt/g8-gw/etcd-data-2/*
etcdq endpoint health
ADD=$(sudo docker run --rm --network g8-gw --user 1000:1000 $IMG \
etcdctl --endpoints=http://etcd-0:2379,http://etcd-1:2379 \
member add etcd-2 --peer-urls=http://etcd-2:2380)
IC=$(echo "$ADD" | grep -oE 'ETCD_INITIAL_CLUSTER="[^"]*"' | sed 's/ETCD_INITIAL_CLUSTER="//; s/"$//')
cat > /tmp/rejoin.yml <<YML
services:
etcd-2:
environment:
ETCD_INITIAL_CLUSTER_STATE: existing
ETCD_INITIAL_CLUSTER: "$IC"
YML
sudo docker compose -f docker-compose.yml -f /tmp/rejoin.yml up -d etcd-2
Once the member has joined, return to the normal definition:
rm -f /tmp/rejoin.yml
sudo docker compose up -d etcd-2
19. Long-term operation
| topic | setting | why |
|---|---|---|
| etcd growth | ETCD_AUTO_COMPACTION_MODE=periodic, RETENTION=1 | Without compaction etcd keeps the old revision of every write; the disk grows without bound |
| etcd quota | ETCD_QUOTA_BACKEND_BYTES=2147483648 | Once the quota is exceeded etcd goes read-only with a NOSPACE alarm. Writing it explicitly stops the alarm from being a surprise |
| WAL truncation | ETCD_SNAPSHOT_COUNT=10000 | A small value raises disk I/O, a large one lengthens recovery |
| Raft timing | HEARTBEAT_INTERVAL=250, ELECTION_TIMEOUT=2500 | These are for a single host. When moving to separate machines, pick a heartbeat of at least ten times the RTT |
| Docker log | max-size: 10m, max-file: 3 | Six containers × 30 MB = 180 MB at most. Covers stdout only (18.4) |
| APISIX file logs | logrotate on the host + docker kill -s USR1 | Outside Docker's quota; set up separately (section 10) |
| File descriptors | nofile 65536 (on all six services) | The Docker default is low; APISIX returns 5xx with too many open files under moderate load |
| Restart | restart: unless-stopped | Docker brings a container back if it dies; stopping by hand is deliberate and unless-stopped respects that |
| Backup | Daily snapshot save, 14 files kept | Section 15 |
That backups are actually being taken must be checked regularly; a timer being
enabled does not prove a file was created:
systemctl list-timers g8-etcd-backup.timer --no-pager
ls -1t /opt/g8-gw/backups/ | head -3
20. Scope limits and next steps
What this installation does not do, stated plainly:
- A single host. Three nodes verify the shape of the configuration, not fault tolerance. If the host dies, all six containers go with it.
- There is no load balancer. The three data plane ports are published separately; the layer that would spread clients across them is out of scope for this document.
- There is no TLS. Traffic between APISIX and etcd, and the Admin API, are
plain HTTP. Network safety rests on
allow_adminand binding the ports to127.0.0.1. TLS is separate work. - Authentication is the admin key only. No users or roles are configured in etcd.
- Restore and upgrade are untested — 15.1 and section 17.
- Behaviour under load was not measured. Every verification in this document is a single request; capacity and latency are separate work.