k3s environment — 3 APISIX + 3 etcd

The host names, addresses and the registry on this page are examples. Our lab run used its own values in their place; replace them with yours, and take the registry address and its credentials from your G8 contact.

Three APISIX Deployment replicas and a three-member etcd StatefulSet on k3s. The manifests are not identical to the OKD ones: everything OKD imposes through its SCC has to be written by hand here.

k3s imposes no restrictions

On OKD the restricted-v2 SCC injects a random uid, runAsNonRoot, capabilities: drop: [ALL], allowPrivilegeEscalation: false and seccompProfile: RuntimeDefault into the pod. k3s provides none of them: the pod runs as the image's own user with the full capability set.

That is why a manifest which is green on k3s says nothing about OKD, and the reverse holds too. The security context lines in this document are not decoration; they are the part that pays for what OKD gives away. The full list of differences is in section 16.

Single node

This installation was verified on a single-node k3s. Three replicas verify the shape of the manifests, not fault tolerance. On a multi-node cluster podAntiAffinity must be required, and that change was not tried here.

1. What we are installing

component	version	source
APISIX	3.19.0-debian	`harbor.g8.dev/infra/apisix:3.19.0-debian`
etcd	v3.5.17	`harbor.g8.dev/infra/etcd:v3.5.17`

object	kind	note
`etcd`	StatefulSet (3) + headless Service	Each member has its own PVC (`local-path`)
`apisix`	Deployment (3) + 3 Services	All three read the same ConfigMap
`apisix`	Ingress (Traefik)	Data plane only; the Admin API is not exposed
`etcd-clients-only`	NetworkPolicy	Decides who may reach etcd
`etcd-backup`	CronJob + PVC	Daily snapshot

2. Prerequisites

requirement	value	note
k3s	v1.36.3+k3s1	The version this was verified on
Storage class	`local-path` (ships with k3s)	`WaitForFirstConsumer`; the PVC binds when the pod is scheduled
Ingress	Traefik (ships with k3s)	`ingressClassName: traefik`
Harbor access	`registries.yaml` or the system CA	Verified below with a real pull

k3s --version | head -1
kubectl get sc --no-headers
kubectl get ingressclass --no-headers

k3s version v1.36.3+k3s1 (5aed4d7b)
local-path (default)   rancher.io/local-path   Delete   WaitForFirstConsumer   false   ...
traefik   traefik.io/ingress-controller   ...

2.1 Ports

port	component	how it is reached
9080	APISIX data plane	`apisix` Service + Ingress (from outside the cluster)
9180	APISIX Admin API	`apisix-admin` Service, in-cluster only. No Ingress
9091	APISIX Prometheus	`apisix-metrics` Service, in-cluster only
9090	APISIX control API	Pod-local `127.0.0.1` only. The probes in section 7 target this; it is never exposed
2379 2380	etcd client and peer	Headless Service, restricted by NetworkPolicy

The Admin API deliberately has no Ingress. If it must be managed from outside the cluster, use kubectl port-forward.

3. Namespace and image pull

Do not skip the pull test: a failure only surfaces as ImagePullBackOff after the StatefulSet is applied, and diagnosing it gets confused with configuration errors. The same command also shows which uid k3s runs the pod as.

kubectl create namespace g8-gateway
kubectl -n g8-gateway create deployment pulltest \
  --image=harbor.g8.dev/infra/apisix:3.19.0-debian -- sleep 120
kubectl -n g8-gateway rollout status deploy/pulltest --timeout=180s
kubectl -n g8-gateway exec deploy/pulltest -- id
kubectl -n g8-gateway delete deploy pulltest

deployment "pulltest" successfully rolled out
uid=636(apisix) gid=636(apisix) groups=636(apisix)

The uid=636(apisix) line matters: k3s uses the image's own user. On OKD the same command prints a number assigned from the namespace range.

4. Security context

The equivalent of what OKD injects through its SCC is written into the manifest by hand here. Two points are critical:

runAsUser must be numeric. The APISIX image declares its user by the name apisix; with runAsNonRoot: true alone, kubelet cannot translate the name to a uid and the pod never starts (15.1).
The etcd image ships with User=0. runAsUser: 1000 keeps it off root and fsGroup: 1000 makes the data directory writable.

These values cannot be carried to OKD: restricted-v2 rejects a uid outside the namespace range. That is why the two platforms' manifests are separate.

5. The etcd cluster

podManagementPolicy: Parallel is mandatory — with the default OrderedReady the cluster never forms (15.2). publishNotReadyAddresses: true is mandatory too: the members must be able to resolve each other while none of them is Ready yet.

The $(POD_NAME) substitution is done by kubelet, not by a shell; the etcd image contains no shell (15.4).

cat <<'EOF' | kubectl apply -f -
apiVersion: v1
kind: Service
metadata:
  name: etcd
  namespace: g8-gateway
spec:
  clusterIP: None
  publishNotReadyAddresses: true
  selector:
    app: etcd
  ports:
    - name: client
      port: 2379
    - name: peer
      port: 2380
---
apiVersion: apps/v1
kind: StatefulSet
metadata:
  name: etcd
  namespace: g8-gateway
spec:
  serviceName: etcd
  podManagementPolicy: Parallel
  replicas: 3
  selector:
    matchLabels:
      app: etcd
  template:
    metadata:
      labels:
        app: etcd
    spec:
      securityContext:
        runAsNonRoot: true
        runAsUser: 1000
        runAsGroup: 1000
        fsGroup: 1000
        seccompProfile:
          type: RuntimeDefault
      affinity:
        podAntiAffinity:
          preferredDuringSchedulingIgnoredDuringExecution:
            - weight: 100
              podAffinityTerm:
                labelSelector:
                  matchLabels:
                    app: etcd
                topologyKey: kubernetes.io/hostname
      containers:
        - name: etcd
          image: harbor.g8.dev/infra/etcd:v3.5.17
          command: ["/usr/local/bin/etcd"]
          securityContext:
            allowPrivilegeEscalation: false
            capabilities:
              drop: ["ALL"]
          env:
            - name: POD_NAME
              valueFrom:
                fieldRef:
                  fieldPath: metadata.name
            - name: ETCD_NAME
              value: "$(POD_NAME)"
            - name: ETCD_DATA_DIR
              value: /etcd-data
            - name: ETCD_LISTEN_PEER_URLS
              value: http://0.0.0.0:2380
            - name: ETCD_LISTEN_CLIENT_URLS
              value: http://0.0.0.0:2379
            - name: ETCD_ADVERTISE_CLIENT_URLS
              value: http://$(POD_NAME).etcd:2379
            - name: ETCD_INITIAL_ADVERTISE_PEER_URLS
              value: http://$(POD_NAME).etcd:2380
            - name: ETCD_INITIAL_CLUSTER
              value: etcd-0=http://etcd-0.etcd:2380,etcd-1=http://etcd-1.etcd:2380,etcd-2=http://etcd-2.etcd:2380
            - name: ETCD_INITIAL_CLUSTER_STATE
              value: new
            - name: ETCD_INITIAL_CLUSTER_TOKEN
              value: g8-k3s
            - name: ETCD_AUTO_COMPACTION_MODE
              value: periodic
            - name: ETCD_AUTO_COMPACTION_RETENTION
              value: "1"
            - name: ETCD_QUOTA_BACKEND_BYTES
              value: "2147483648"
            - name: ETCD_SNAPSHOT_COUNT
              value: "10000"
            - name: ETCD_HEARTBEAT_INTERVAL
              value: "250"
            - name: ETCD_ELECTION_TIMEOUT
              value: "2500"
          ports:
            - name: client
              containerPort: 2379
            - name: peer
              containerPort: 2380
          readinessProbe:
            httpGet:
              path: /readyz
              port: 2379
            initialDelaySeconds: 10
            periodSeconds: 10
            timeoutSeconds: 10
          livenessProbe:
            httpGet:
              path: /livez
              port: 2379
            initialDelaySeconds: 30
            periodSeconds: 15
            timeoutSeconds: 10
          resources:
            requests:
              cpu: 100m
              memory: 256Mi
            limits:
              cpu: "1"
              memory: 1Gi
          volumeMounts:
            - name: data
              mountPath: /etcd-data
  volumeClaimTemplates:
    - metadata:
        name: data
      spec:
        accessModes: [ReadWriteOnce]
        storageClassName: local-path
        resources:
          requests:
            storage: 8Gi
EOF

kubectl -n g8-gateway rollout status sts/etcd --timeout=300s

local-path does not support volume expansion — pick the size now

kubectl get sc local-path -o jsonpath='{.allowVolumeExpansion}' comes back empty: these PVCs cannot be grown later. The only way to enlarge one is to delete and recreate it, which erases that member's data and requires the rejoin procedure in 15.8.

The 8Gi above is therefore a decision made at install time. The quota is 2 GB; the margin covers the WAL files (which sit outside the quota) and the space defragmentation temporarily needs — defrag wants free space equal to the database size.

OKD is different: topolvm-provisioner supports expansion and the PVC can be grown in place.

6. Admin key

APISIX's documented default admin key (edd1c9f0…) is public knowledge. The key is kept in a Secret and passed to the pod as an environment variable.

kubectl -n g8-gateway create secret generic apisix-admin-key \
  --from-literal=G8_APISIX_ADMIN_KEY="$(openssl rand -hex 24)"

7. APISIX and Ingress

Logs go to stdout/stderr, not to files; kubelet collects and rotates them (section 17). The probes target the control port 9090, not 9080; a bare TCP connection to 9080 puts a "status":"400" line into the access log.

allow_admin covers the pod network (10.42.0.0/16) and loopback. k3s's hairpin SNAT address (10.42.0.1) is inside the pod network, so the extra entry needed on OKD is not needed here (15.3).

cat <<'EOF' | kubectl apply -f -
apiVersion: v1
kind: ConfigMap
metadata:
  name: apisix-config
  namespace: g8-gateway
data:
  config.yaml: |
    apisix:
      node_listen:
        - 9080

    deployment:
      role: traditional
      role_traditional:
        config_provider: etcd
      admin:
        admin_listen:
          ip: 0.0.0.0
          port: 9180
        admin_key:
          - name: admin
            key: "${{G8_APISIX_ADMIN_KEY}}"
            role: admin
        allow_admin:
          - 10.42.0.0/16
          - 127.0.0.0/8
      etcd:
        host:
          - "http://etcd-0.etcd:2379"
          - "http://etcd-1.etcd:2379"
          - "http://etcd-2.etcd:2379"
        prefix: /apisix
        timeout: 30

    nginx_config:
      error_log: /dev/stderr
      error_log_level: warn
      http:
        access_log: /dev/stdout
        access_log_format: '{"time":"$time_iso8601","remote_addr":"$remote_addr","status":"$status","method":"$request_method","uri":"$uri","rt":"$request_time","ua":"$http_user_agent"}'
        access_log_format_escape: json

    plugin_attr:
      prometheus:
        export_addr:
          ip: 0.0.0.0
          port: 9091
        metrics:
          http_status:
            extra_labels:
              - route_name: $route_name
              - service_name: $service_name
          http_latency:
            extra_labels:
              - route_name: $route_name
              - service_name: $service_name
---
apiVersion: apps/v1
kind: Deployment
metadata:
  name: apisix
  namespace: g8-gateway
spec:
  replicas: 3
  selector:
    matchLabels:
      app: apisix
  template:
    metadata:
      labels:
        app: apisix
    spec:
      securityContext:
        runAsNonRoot: true
        runAsUser: 636
        runAsGroup: 636
        seccompProfile:
          type: RuntimeDefault
      affinity:
        podAntiAffinity:
          preferredDuringSchedulingIgnoredDuringExecution:
            - weight: 100
              podAffinityTerm:
                labelSelector:
                  matchLabels:
                    app: apisix
                topologyKey: kubernetes.io/hostname
      containers:
        - name: apisix
          image: harbor.g8.dev/infra/apisix:3.19.0-debian
          securityContext:
            allowPrivilegeEscalation: false
            capabilities:
              drop: ["ALL"]
          envFrom:
            - secretRef:
                name: apisix-admin-key
          ports:
            - name: http
              containerPort: 9080
            - name: admin
              containerPort: 9180
            - name: metrics
              containerPort: 9091
          readinessProbe:
            exec:
              command: ["bash", "-c", "echo > /dev/tcp/127.0.0.1/9090"]
            initialDelaySeconds: 10
            periodSeconds: 10
            timeoutSeconds: 5
          livenessProbe:
            exec:
              command: ["bash", "-c", "echo > /dev/tcp/127.0.0.1/9090"]
            initialDelaySeconds: 30
            periodSeconds: 15
            timeoutSeconds: 5
          resources:
            requests:
              cpu: 100m
              memory: 256Mi
            limits:
              cpu: "2"
              memory: 1Gi
          volumeMounts:
            - name: config
              mountPath: /usr/local/apisix/conf/config.yaml
              subPath: config.yaml
              readOnly: true
      volumes:
        - name: config
          configMap:
            name: apisix-config
---
apiVersion: v1
kind: Service
metadata:
  name: apisix
  namespace: g8-gateway
spec:
  selector:
    app: apisix
  ports:
    - name: http
      port: 9080
      targetPort: 9080
---
apiVersion: v1
kind: Service
metadata:
  name: apisix-admin
  namespace: g8-gateway
spec:
  selector:
    app: apisix
  ports:
    - name: admin
      port: 9180
      targetPort: 9180
---
apiVersion: v1
kind: Service
metadata:
  name: apisix-metrics
  namespace: g8-gateway
spec:
  selector:
    app: apisix
  ports:
    - name: metrics
      port: 9091
      targetPort: 9091
---
apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
  name: apisix
  namespace: g8-gateway
spec:
  ingressClassName: traefik
  rules:
    - host: gw.k3s.example.com
      http:
        paths:
          - path: /
            pathType: Prefix
            backend:
              service:
                name: apisix
                port:
                  number: 9080
EOF

kubectl -n g8-gateway rollout status deploy/apisix --timeout=300s

The host: value in the Ingress must be changed to match your own DNS. If no record resolves the name, it can also be tested with curl -H "Host: gw.k3s.example.com" http://<node-ip>/smoke.

7.1 PodDisruptionBudget

When Kubernetes drains a node (maintenance, upgrade, autoscaling) it evicts the pods. By default there is no limit: when measured, all three etcd pods could be evicted back to back and each returned 201 Success — meaning a single maintenance operation can take quorum with it.

A PodDisruptionBudget stops that at the API level:

cat <<'EOF' | kubectl apply -f -
apiVersion: policy/v1
kind: PodDisruptionBudget
metadata:
  name: etcd
  namespace: g8-gateway
spec:
  minAvailable: 2
  selector:
    matchLabels:
      app: etcd
---
apiVersion: policy/v1
kind: PodDisruptionBudget
metadata:
  name: apisix
  namespace: g8-gateway
spec:
  minAvailable: 2
  selector:
    matchLabels:
      app: apisix
EOF

Verification:

kubectl -n g8-gateway get pdb

NAME     MIN AVAILABLE   MAX UNAVAILABLE   ALLOWED DISRUPTIONS
apisix   2               N/A               1
etcd     2               N/A               1

The ALLOWED DISRUPTIONS column is the real answer: one pod may be evicted, a second may not. Measured:

1. tahliye (etcd-0): 201 Success        -> izinli kesinti 1 -> 0
2. tahliye (etcd-1): TooManyRequests: Cannot evict pod as it
                     would violate the pod's disruption budget

This blocks voluntary evictions; if a node crashes, the PDB does not come into play. What it protects against is maintenance-induced outage.

8. NetworkPolicy

etcd has no authentication. Without a NetworkPolicy, any pod in the cluster can read and rewrite the gateway's entire configuration.

k3s enforces NetworkPolicy by default, and that is verified (10.6). Writing the policy without measuring that it is enforced means being silently unprotected on a CNI that ignores it.

cat <<'EOF' | kubectl apply -f -
apiVersion: networking.k8s.io/v1
kind: NetworkPolicy
metadata:
  name: etcd-clients-only
  namespace: g8-gateway
spec:
  podSelector:
    matchLabels:
      app: etcd
  policyTypes: [Ingress]
  ingress:
    - from:
        - podSelector:
            matchLabels:
              app: apisix
        - podSelector:
            matchLabels:
              app: etcd
        - podSelector:
            matchLabels:
              app: etcd-backup
      ports:
        - protocol: TCP
          port: 2379
        - protocol: TCP
          port: 2380
EOF

9. Backup

The snapshot is written to a separate PVC. The filename is made unique with $(POD_NAME) — date cannot be called because the etcd image has no shell. Pruning old files falls to an initContainer that does have a shell.

cat <<'EOF' | kubectl apply -f -
apiVersion: v1
kind: PersistentVolumeClaim
metadata:
  name: etcd-backup
  namespace: g8-gateway
spec:
  accessModes: [ReadWriteOnce]
  storageClassName: local-path
  resources:
    requests:
      storage: 5Gi
---
apiVersion: batch/v1
kind: CronJob
metadata:
  name: etcd-backup
  namespace: g8-gateway
spec:
  schedule: "0 2 * * *"
  concurrencyPolicy: Forbid
  successfulJobsHistoryLimit: 3
  failedJobsHistoryLimit: 3
  jobTemplate:
    spec:
      backoffLimit: 2
      activeDeadlineSeconds: 1800
      template:
        metadata:
          labels:
            app: etcd-backup
        spec:
          restartPolicy: Never
          securityContext:
            runAsNonRoot: true
            runAsUser: 1000
            runAsGroup: 1000
            fsGroup: 1000
            seccompProfile:
              type: RuntimeDefault
          initContainers:
            - name: prune
              image: harbor.g8.dev/infra/apisix:3.19.0-debian
              securityContext:
                allowPrivilegeEscalation: false
                capabilities:
                  drop: ["ALL"]
              command:
                - bash
                - -c
                - "ls -1t /backups/etcd-*.db 2>/dev/null | tail -n +15 | xargs -r rm -f; echo pruned"
              volumeMounts:
                - name: backups
                  mountPath: /backups
          containers:
            - name: snapshot
              image: harbor.g8.dev/infra/etcd:v3.5.17
              securityContext:
                allowPrivilegeEscalation: false
                capabilities:
                  drop: ["ALL"]
              command: ["/usr/local/bin/etcdctl"]
              args:
                - "--endpoints=http://etcd-0.etcd:2379"
                - "snapshot"
                - "save"
                - "/backups/etcd-$(POD_NAME).db"
              env:
                - name: POD_NAME
                  valueFrom:
                    fieldRef:
                      fieldPath: metadata.name
              volumeMounts:
                - name: backups
                  mountPath: /backups
          volumes:
            - name: backups
              persistentVolumeClaim:
                claimName: etcd-backup
EOF

snapshot save requires a single endpoint; passing several returns snapshot must be requested to one selected node.

The backups sit on the same node as the data they protect

The etcd-backup PVC is on the same node, on the same disk, as the etcd data itself. If the node or the disk goes, the backup goes with it — so this backup does not protect against the event it promises to.

This installation does not move backups off the cluster; a customer environment must. The direction matters: the remote system pulls, and the gateway should hold no write credential on it — anyone who takes the gateway would use that credential to delete the backups too. A working example for the virtual-machine installs is in 02-test-vm §13.3.

9.1 Defragmentation

Compaction removes old revisions logically but does not shrink the database file — the freed pages stay inside etcd and never return to the filesystem. Since the quota watches dbSize, the file keeps growing even when compaction is working perfectly, and the quota fills sooner or later. The extreme measured: dbSizeInUse at 41 kB while dbSize sat at 98 MB.

Defragmentation therefore has to be scheduled. Weekly is enough.

cat <<'EOF' | kubectl apply -f -
apiVersion: batch/v1
kind: CronJob
metadata:
  name: etcd-defrag
  namespace: g8-gateway
spec:
  schedule: "0 3 * * 0"
  concurrencyPolicy: Forbid
  successfulJobsHistoryLimit: 3
  failedJobsHistoryLimit: 3
  jobTemplate:
    spec:
      backoffLimit: 1
      activeDeadlineSeconds: 3600
      template:
        metadata:
          labels:
            app: etcd-backup
        spec:
          restartPolicy: Never
          securityContext:
            runAsNonRoot: true
            runAsUser: 1000
            runAsGroup: 1000
            seccompProfile:
              type: RuntimeDefault
          initContainers:
            - name: defrag-0
              image: harbor.g8.dev/infra/etcd:v3.5.17
              command: ["/usr/local/bin/etcdctl"]
              args: ["--endpoints=http://etcd-0.etcd:2379", "--command-timeout=900s", "defrag"]
            - name: defrag-1
              image: harbor.g8.dev/infra/etcd:v3.5.17
              command: ["/usr/local/bin/etcdctl"]
              args: ["--endpoints=http://etcd-1.etcd:2379", "--command-timeout=900s", "defrag"]
            - name: defrag-2
              image: harbor.g8.dev/infra/etcd:v3.5.17
              command: ["/usr/local/bin/etcdctl"]
              args: ["--endpoints=http://etcd-2.etcd:2379", "--command-timeout=900s", "defrag"]
          containers:
            - name: alarms
              image: harbor.g8.dev/infra/etcd:v3.5.17
              command: ["/usr/local/bin/etcdctl"]
              args: ["--endpoints=http://etcd-0.etcd:2379,http://etcd-1.etcd:2379,http://etcd-2.etcd:2379", "--command-timeout=30s", "alarm", "list"]
EOF

Verification:

kubectl -n g8-gateway create job defrag-now --from=cronjob/etcd-defrag
kubectl -n g8-gateway wait --for=condition=complete job/defrag-now --timeout=600s
POD=$(kubectl -n g8-gateway get pod -l job-name=defrag-now -o jsonpath='{.items[0].metadata.name}')
for c in defrag-0 defrag-1 defrag-2; do kubectl -n g8-gateway logs $POD -c $c | tail -1; done
kubectl -n g8-gateway delete job defrag-now

Finished defragmenting etcd member[http://etcd-0.etcd:2379]
Finished defragmenting etcd member[http://etcd-1.etcd:2379]
Finished defragmenting etcd member[http://etcd-2.etcd:2379]

Defrag blocks that member briefly, which is why the members are processed one at a time. Defragmenting all three at once drops quorum.

Defrag temporarily needs free space equal to the database size; that is one reason the quota must be meaningfully smaller than the disk.

10. Verification

10.1 Pods and the security context

That the security context was really applied is read from the pod definition. Unlike OKD, the values here are the ones we wrote; if they look empty, the manifest was not applied.

kubectl -n g8-gateway get pod -o custom-columns=\
'POD:.metadata.name,UID:.spec.securityContext.runAsUser,NONROOT:.spec.securityContext.runAsNonRoot' \
  --no-headers | sort

apisix-85b7788557-2jnxx   636    true
apisix-85b7788557-96xhl   636    true
apisix-85b7788557-llng4   636    true
etcd-0                    1000   true
etcd-1                    1000   true
etcd-2                    1000   true

10.2 The etcd cluster

The member count, the healthy-endpoint count and the leader count are each counted separately.

E=http://etcd-0.etcd:2379,http://etcd-1.etcd:2379,http://etcd-2.etcd:2379
echo "members=$(kubectl -n g8-gateway exec etcd-0 -- etcdctl --endpoints=$E member list -w simple | wc -l)"
echo "healthy=$(kubectl -n g8-gateway exec etcd-0 -- etcdctl --endpoints=$E endpoint health -w simple 2>&1 | grep -c 'is healthy')"
echo "leaders=$(kubectl -n g8-gateway exec etcd-0 -- etcdctl --endpoints=$E endpoint status -w simple | awk -F, '$5 ~ /true/' | wc -l)"

members=3
healthy=3
leaders=1

10.3 Admin API and key substitution

The Admin API has no Ingress; access is opened with kubectl port-forward. Three requests: the real key, the unsubstituted raw template, and no key.

kubectl -n g8-gateway port-forward svc/apisix-admin 19180:9180 >/dev/null 2>&1 &
PF=$!
sleep 6
KEY=$(kubectl -n g8-gateway get secret apisix-admin-key -o jsonpath='{.data.G8_APISIX_ADMIN_KEY}' | base64 -d)
umask 077
printf 'header = "X-API-KEY: %s"\n' "$KEY" > /tmp/g8adm.conf
printf 'header = "X-API-KEY: ${{G8_APISIX_ADMIN_KEY}}"\n' > /tmp/g8raw.conf
curl -s -o /dev/null -w "real=%{http_code}\n" -K /tmp/g8adm.conf http://127.0.0.1:19180/apisix/admin/routes
curl -s -o /dev/null -w "raw=%{http_code}\n"  -K /tmp/g8raw.conf http://127.0.0.1:19180/apisix/admin/routes
curl -s -o /dev/null -w "none=%{http_code}\n"                    http://127.0.0.1:19180/apisix/admin/routes

real=200
raw=401
none=401

10.4 continues in this shell session

The next section uses the $PF variable set here, the /tmp/g8adm.conf file, and the port-forward that is still running. Open a new terminal and 10.4 fails with a connection error. Stay in the same shell.

10.4 Does the configuration propagate to all three pods

The route is written once and read back from each of the three pods separately. Reading through the Ingress is not enough: if load balancing lands on the same pod, the other two are never tested.

curl -s -o /dev/null -w "PUT=%{http_code}\n" -K /tmp/g8adm.conf \
  -X PUT http://127.0.0.1:19180/apisix/admin/routes/g8-smoke \
  -H "Content-Type: application/json" \
  -d '{"uri":"/smoke","plugins":{"mocking":{"content_type":"text/plain","response_status":200,"response_example":"G8-GW-K3S-OK"}}}'
kill $PF 2>/dev/null
sleep 3
for p in $(kubectl -n g8-gateway get pod -l app=apisix -o jsonpath='{.items[*].metadata.name}'); do
  kubectl -n g8-gateway port-forward pod/$p 19081:9080 >/dev/null 2>&1 &
  PF=$!
  sleep 4
  echo "$(curl -s -o /dev/null -w '%{http_code}' http://127.0.0.1:19081/smoke) $(curl -s http://127.0.0.1:19081/smoke)"
  kill $PF 2>/dev/null
  sleep 1
done

PUT=201
G8-GW-K3S-OK
G8-GW-K3S-OK
G8-GW-K3S-OK

10.5 The Ingress and the source IP

Measured deviation, 2026-09-30

The lab's k3s cluster carries one object this document does not: an apisix-admin Ingress that publishes the Admin API through Traefik, so that g8 on the OKD cluster can manage this environment. This document publishes only the data plane; publish the Admin API only for a management host, with the same allow_admin discipline as the Docker installations.

curl -s http://gw.k3s.example.com/smoke -w "\nstatus=%{http_code}\n"

G8-GW-K3S-OK
status=200

Which address the request arrives at APISIX with is read from the access log:

curl -s -o /dev/null http://gw.k3s.example.com/smoke
sleep 8
kubectl -n g8-gateway logs -l app=apisix --tail=50 --prefix=false \
  | grep '"uri":"/smoke"' | sort | tail -1
kubectl -n kube-system get pod -l app.kubernetes.io/name=traefik \
  -o jsonpath='{.items[0].status.podIP}{"\n"}'

{"time":"2026-08-24T21:58:18+00:00","remote_addr":"10.42.0.8","status":"200","method":"GET","uri":"/smoke","rt":"0.000","ua":"curl/8.5.0"}
10.42.0.8

remote_addr is not the client's address; it is the Traefik pod's. If the real client IP is needed, Traefik's X-Forwarded-For header and the real-ip plugin in APISIX must be configured — this document does not cover that.

The sort is mandatory: kubectl logs -l concatenates the logs of several pods and the order is not chronological (15.5).

10.6 NetworkPolicy

That the policy works is measured with both controls. Seeing "blocked" alone is not enough: the pod may simply never have started.

cat <<'EOF' | kubectl apply -f -
apiVersion: v1
kind: Pod
metadata:
  name: npdeny
  namespace: g8-gateway
  labels:
    app: outsider
spec:
  restartPolicy: Never
  securityContext:
    runAsNonRoot: true
    runAsUser: 636
    runAsGroup: 636
  containers:
    - name: t
      image: harbor.g8.dev/infra/apisix:3.19.0-debian
      command: ["bash","-c","timeout 8 bash -c 'echo > /dev/tcp/etcd-0.etcd/2379' && echo REACHABLE || echo BLOCKED"]
EOF
kubectl -n g8-gateway wait --for=jsonpath='{.status.phase}'=Succeeded pod/npdeny --timeout=120s >/dev/null
echo "unlabelled=$(kubectl -n g8-gateway logs npdeny | tail -1)"
kubectl -n g8-gateway delete pod npdeny
A=$(kubectl -n g8-gateway get pod -l app=apisix -o jsonpath='{.items[0].metadata.name}')
echo "apisix=$(kubectl -n g8-gateway exec $A -- bash -c "timeout 8 bash -c 'echo > /dev/tcp/etcd-0.etcd/2379' && echo REACHABLE || echo BLOCKED")"

unlabelled=BLOCKED
apisix=REACHABLE

On k3s a blocked connection comes back as Connection refused (an active reject); on OKD it is dropped silently and times out. The symptom differs, the outcome is the same.

10.7 Quorum: one member down

The leader pod is deleted. Expected: the cluster elects a new leader, writes continue, the data plane is unaffected, and the StatefulSet brings the pod back.

E=http://etcd-0.etcd:2379,http://etcd-1.etcd:2379,http://etcd-2.etcd:2379
LEADER=$(kubectl -n g8-gateway exec etcd-0 -- etcdctl --endpoints=$E endpoint status -w simple \
  | awk -F, '$5 ~ /true/ {print $1}' | sed 's|http://||;s|\.etcd:2379||')
kubectl -n g8-gateway delete pod $LEADER --wait=false
sleep 15
S=$(kubectl -n g8-gateway get pod -l app=etcd --no-headers | grep '1/1' | head -1 | awk '{print $1}')
kubectl -n g8-gateway exec $S -- etcdctl --endpoints=http://127.0.0.1:2379 put /g8-qp ok
curl -s http://gw.k3s.example.com/smoke -w "\nstatus=%{http_code}\n"
kubectl -n g8-gateway rollout status sts/etcd --timeout=300s
kubectl -n g8-gateway exec etcd-2 -- etcdctl --endpoints=$E get /g8-qp --print-value-only
kubectl -n g8-gateway exec etcd-0 -- etcdctl --endpoints=$E del /g8-qp

OK
G8-GW-K3S-OK
status=200
partitioned roll out complete: 3 new pods have been updated...
ok
1

10.8 Quorum loss

Scaling the members down to one leaves no quorum. The expected behaviour has two parts and both must be measured: the write fails, and the data plane keeps serving the existing routes.

kubectl -n g8-gateway scale sts etcd --replicas=1
sleep 40
if kubectl -n g8-gateway exec etcd-0 -- etcdctl --endpoints=http://127.0.0.1:2379 --command-timeout=15s put /g8-noquorum x >/dev/null 2>&1
then echo "write=UNEXPECTED-SUCCESS"; else echo "write=FAILED"; fi
curl -s -o /dev/null --max-time 10 -w "ingress=%{http_code}\n" http://gw.k3s.example.com/smoke
echo "apisix_ready=$(kubectl -n g8-gateway get pod -l app=apisix --no-headers | grep -c '1/1')"
echo "etcd_restarts=$(kubectl -n g8-gateway get pod etcd-0 -o jsonpath='{.status.containerStatuses[0].restartCount}')"

write=FAILED
ingress=200
apisix_ready=3
etcd_restarts=0

No error string is printed here: the string etcd returns varies from run to run — three independent measurements produced three different ones. What is asserted instead is that the write failed, which is the actual question.

etcd_restarts=0 is a regression sentinel. Anything above zero means the liveness probe is watching cluster health and killing the last member on quorum loss (15.6).

The APISIX pods stay ready because their probes target the control port, not etcd.

The cluster is brought back:

kubectl -n g8-gateway scale sts etcd --replicas=3
kubectl -n g8-gateway rollout status sts/etcd --timeout=300s
E=http://etcd-0.etcd:2379,http://etcd-1.etcd:2379,http://etcd-2.etcd:2379
echo "healthy=$(kubectl -n g8-gateway exec etcd-0 -- etcdctl --endpoints=$E endpoint health -w simple 2>&1 | grep -c 'is healthy')"

partitioned roll out complete: 3 new pods have been updated...
healthy=3

How the Admin API behaves meanwhile was measured separately:

healthy: PUT=201 time=0.074334s
req1:    PUT=503 time=42.471697s
req2:    PUT=503 time=0.038192s
req3:    PUT=503 time=0.020874s

The first write hangs for tens of seconds; later ones return 503 instantly (has no healthy etcd endpoint available). A fast 503 does not mean etcd is back. The behaviour differs in the prod (Docker) installation — there the requests do not get faster (03-prod-docker, 17.2).

10.9 Is the backup valid

That the CronJob exists does not prove a backup was taken. It is run once by hand and the file it produces is verified.

kubectl -n g8-gateway create job manual-backup --from=cronjob/etcd-backup
kubectl -n g8-gateway wait --for=condition=complete job/manual-backup --timeout=300s
POD=$(kubectl -n g8-gateway get pod -l job-name=manual-backup -o jsonpath='{.items[0].metadata.name}')
kubectl -n g8-gateway logs $POD -c prune
kubectl -n g8-gateway logs $POD -c snapshot 2>&1 | grep -o '/backups/[^"]*\.db' | tail -1
kubectl -n g8-gateway delete job manual-backup

pruned
/backups/etcd-manual-backup-sjzrs.db

The filename varies per job run. That the file really is a readable snapshot is tested separately — replace the name below with the one from your own output:

cat <<'EOF' | kubectl apply -f -
apiVersion: v1
kind: Pod
metadata:
  name: snapcheck
  namespace: g8-gateway
  labels:
    app: etcd-backup
spec:
  restartPolicy: Never
  securityContext:
    runAsNonRoot: true
    runAsUser: 1000
    runAsGroup: 1000
    fsGroup: 1000
  containers:
    - name: c
      image: harbor.g8.dev/infra/etcd:v3.5.17
      command: ["/usr/local/bin/etcdutl"]
      args: ["snapshot","status","/backups/etcd-manual-backup-sjzrs.db","-w","table"]
      volumeMounts:
        - name: backups
          mountPath: /backups
  volumes:
    - name: backups
      persistentVolumeClaim:
        claimName: etcd-backup
EOF
kubectl -n g8-gateway wait --for=jsonpath='{.status.phase}'=Succeeded pod/snapcheck --timeout=180s >/dev/null
kubectl -n g8-gateway logs snapcheck
kubectl -n g8-gateway delete pod snapcheck

+----------+----------+------------+------------+
|   HASH   | REVISION | TOTAL KEYS | TOTAL SIZE |
+----------+----------+------------+------------+
| e417a46a |       45 |         52 |      25 kB |
+----------+----------+------------+------------+

10.10 Cleaning up verification leftovers

10.3 writes the admin key into /tmp/g8adm.conf in cleartext. If the file is not deleted the key stays on disk, and the care section 6 takes to keep the key in a Secret is wasted here.

rm -f /tmp/g8adm.conf /tmp/g8raw.conf
ls /tmp/g8*.conf 2>/dev/null | wc -l

The g8-smoke route is left in place deliberately: it is lasting proof that the installation works, and the uninstall in section 13 removes it along with the namespace. On a cluster that will carry production traffic, delete it through the Admin API if you do not want it there.

11. Observability

This environment gets its own Elasticsearch and Prometheus, both running inside the cluster. Request records go to Elasticsearch, request metrics to Prometheus.

component	version	source
Elasticsearch	9.5.2	`harbor.g8.dev/infra/elasticsearch:9.5.2`
Prometheus	3.15.0	`harbor.g8.dev/infra/prometheus:v3.15.0`

Prometheus starts with --enable-feature=promql-extended-range-selectors: Analytics counts each bucket exactly with the anchored range selector, which Prometheus 3.15 takes only behind that flag. From Prometheus 3.16 the selector is on by default and the line can go; without it, Analytics shows its counts as estimates.

Prometheus finds the APISIX pods through Kubernetes service discovery, so the configuration does not change when the pod count does. external_labels marks the environment.

11.1 Elasticsearch

observability-es.yaml
apiVersion: v1
kind: Service
metadata:
  name: elasticsearch
  namespace: g8-gateway
spec:
  selector:
    app: elasticsearch
  ports:
    - name: http
      port: 9200
      targetPort: 9200
---
apiVersion: apps/v1
kind: StatefulSet
metadata:
  name: elasticsearch
  namespace: g8-gateway
spec:
  serviceName: elasticsearch
  replicas: 1
  selector:
    matchLabels:
      app: elasticsearch
  template:
    metadata:
      labels:
        app: elasticsearch
    spec:
      securityContext:
        runAsNonRoot: true
        runAsUser: 1000
        runAsGroup: 1000
        fsGroup: 1000
        seccompProfile:
          type: RuntimeDefault
      containers:
        - name: elasticsearch
          image: harbor.g8.dev/infra/elasticsearch:9.5.2
          securityContext:
            allowPrivilegeEscalation: false
            capabilities:
              drop: ["ALL"]
          env:
            - name: node.name
              value: es-k3s
            - name: cluster.name
              value: g8-k3s
            - name: discovery.type
              value: single-node
            - name: ES_JAVA_OPTS
              value: "-Xms1g -Xmx1g"
            - name: xpack.security.enabled
              value: "true"
            - name: xpack.security.enrollment.enabled
              value: "false"
            - name: xpack.security.http.ssl.enabled
              value: "false"
            - name: xpack.security.transport.ssl.enabled
              value: "false"
            - name: ingest.geoip.downloader.enabled
              value: "false"
            - name: bootstrap.memory_lock
              value: "false"
            - name: ELASTIC_PASSWORD
              valueFrom:
                secretKeyRef:
                  name: g8-es-credentials
                  key: password
          ports:
            - name: http
              containerPort: 9200
          resources:
            requests:
              cpu: 300m
              memory: 1600Mi
            limits:
              cpu: "2"
              memory: 2200Mi
          readinessProbe:
            exec:
              command:
                - sh
                - -c
                - curl -sf -u "elastic:$ELASTIC_PASSWORD" http://127.0.0.1:9200/_cluster/health?local=true >/dev/null
            initialDelaySeconds: 60
            periodSeconds: 15
            failureThreshold: 20
          livenessProbe:
            tcpSocket:
              port: 9200
            initialDelaySeconds: 120
            periodSeconds: 20
            failureThreshold: 6
          volumeMounts:
            - name: data
              mountPath: /usr/share/elasticsearch/data
  volumeClaimTemplates:
    - metadata:
        name: data
      spec:
        accessModes: ["ReadWriteOnce"]
        storageClassName: local-path
        resources:
          requests:
            storage: 20Gi

11.2 Prometheus and NetworkPolicy

observability-prometheus.yaml
apiVersion: v1
kind: ConfigMap
metadata:
  name: prometheus-config
  namespace: g8-gateway
data:
  prometheus.yml: |
    global:
      scrape_interval: 15s
      evaluation_interval: 15s
      external_labels:
        g8_environment: k3s
    scrape_configs:
      - job_name: apisix
        metrics_path: /apisix/prometheus/metrics
        kubernetes_sd_configs:
          - role: pod
            namespaces:
              names: [g8-gateway]
        relabel_configs:
          - source_labels: [__meta_kubernetes_pod_label_app]
            action: keep
            regex: apisix
          - source_labels: [__address__]
            action: replace
            regex: '([^:]+)(:\d+)?'
            replacement: '$1:9091'
            target_label: __address__
          - source_labels: [__meta_kubernetes_pod_name]
            target_label: pod
          - target_label: g8_environment
            replacement: k3s
      - job_name: prometheus
        static_configs:
          - targets: ["127.0.0.1:9090"]
---
apiVersion: v1
kind: ServiceAccount
metadata:
  name: prometheus
  namespace: g8-gateway
---
apiVersion: rbac.authorization.k8s.io/v1
kind: Role
metadata:
  name: prometheus
  namespace: g8-gateway
rules:
  - apiGroups: [""]
    resources: ["pods", "services", "endpoints"]
    verbs: ["get", "list", "watch"]
---
apiVersion: rbac.authorization.k8s.io/v1
kind: RoleBinding
metadata:
  name: prometheus
  namespace: g8-gateway
roleRef:
  apiGroup: rbac.authorization.k8s.io
  kind: Role
  name: prometheus
subjects:
  - kind: ServiceAccount
    name: prometheus
    namespace: g8-gateway
---
apiVersion: v1
kind: PersistentVolumeClaim
metadata:
  name: prometheus-data
  namespace: g8-gateway
spec:
  accessModes: ["ReadWriteOnce"]
  storageClassName: local-path
  resources:
    requests:
      storage: 10Gi
---
apiVersion: v1
kind: Service
metadata:
  name: prometheus
  namespace: g8-gateway
spec:
  selector:
    app: prometheus
  ports:
    - name: http
      port: 9090
      targetPort: 9090
---
apiVersion: apps/v1
kind: Deployment
metadata:
  name: prometheus
  namespace: g8-gateway
spec:
  replicas: 1
  strategy:
    type: Recreate
  selector:
    matchLabels:
      app: prometheus
  template:
    metadata:
      labels:
        app: prometheus
    spec:
      serviceAccountName: prometheus
      securityContext:
        runAsNonRoot: true
        runAsUser: 65534
        runAsGroup: 65534
        fsGroup: 65534
        seccompProfile:
          type: RuntimeDefault
      containers:
        - name: prometheus
          image: harbor.g8.dev/infra/prometheus:v3.15.0
          securityContext:
            allowPrivilegeEscalation: false
            capabilities:
              drop: ["ALL"]
          args:
            - --config.file=/etc/prometheus/prometheus.yml
            - --storage.tsdb.path=/prometheus
            - --storage.tsdb.retention.time=40d
            - --storage.tsdb.retention.size=5GB
            - --enable-feature=promql-extended-range-selectors
          ports:
            - name: http
              containerPort: 9090
          resources:
            requests:
              cpu: 100m
              memory: 256Mi
            limits:
              cpu: "1"
              memory: 1Gi
          readinessProbe:
            httpGet:
              path: /-/ready
              port: 9090
            initialDelaySeconds: 10
            periodSeconds: 10
          volumeMounts:
            - name: config
              mountPath: /etc/prometheus
            - name: data
              mountPath: /prometheus
      volumes:
        - name: config
          configMap:
            name: prometheus-config
        - name: data
          persistentVolumeClaim:
            claimName: prometheus-data
---
apiVersion: networking.k8s.io/v1
kind: NetworkPolicy
metadata:
  name: elasticsearch-clients-only
  namespace: g8-gateway
spec:
  podSelector:
    matchLabels:
      app: elasticsearch
  policyTypes: ["Ingress"]
  ingress:
    - from:
        - podSelector:
            matchLabels:
              app: apisix
        - podSelector:
            matchLabels:
              app: prometheus
        - namespaceSelector:
            matchLabels:
              kubernetes.io/metadata.name: kube-system
      ports:
        - protocol: TCP
          port: 9200

The NetworkPolicy lets only APISIX, Prometheus and the ingress controller reach Elasticsearch.

The ingress controller's namespace must be in the NetworkPolicy

Leave kube-system out of the list and the Route/Ingress looks healthy while requests never reach Elasticsearch. The symptom is a timeout, not a clear rejection.

11.3 Exposing

observability-expose.yaml
apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
  name: elasticsearch
  namespace: g8-gateway
spec:
  ingressClassName: traefik
  rules:
    - host: es.k3s.example.com
      http:
        paths:
          - path: /
            pathType: Prefix
            backend:
              service:
                name: elasticsearch
                port:
                  number: 9200
---
apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
  name: prometheus
  namespace: g8-gateway
spec:
  ingressClassName: traefik
  rules:
    - host: prom.k3s.example.com
      http:
        paths:
          - path: /
            pathType: Prefix
            backend:
              service:
                name: prometheus
                port:
                  number: 9090

11.4 Install

The password is generated in the cluster and reaches neither this document nor the repository. It is the store administrator's password and no rule of the gateway carries it: the gateway writes with an account of its own (11.6).

kubectl -n g8-gateway create secret generic g8-es-credentials \
  --from-literal=password="$(openssl rand -base64 24 | tr -d /+= | head -c 24)"

kubectl apply -f observability-es.yaml
kubectl apply -f observability-prometheus.yaml
kubectl apply -f observability-expose.yaml

kubectl -n g8-gateway wait --for=condition=Ready pod/elasticsearch-0 --timeout=300s
kubectl -n g8-gateway rollout status deploy/prometheus --timeout=300s

Wait for readiness with wait

After delete pod --wait=false a loop that polls readiness can read the stale pod object, call it ready and exit early. One verification reported data loss because of exactly this; the real cause was that the pod had not come up yet.

11.5 Wiring APISIX

The metrics are enabled through a global rule, which leaves prefer_name off: route and service on the metrics carry the ids Analytics counts by, and metrics.*.extra_labels in config.yaml put each route's and service's name beside them as route_name and service_name. The request records are not wired here: g8 writes their rule itself when traffic logging is turned on (11.7).

KEY=$(kubectl -n g8-gateway get secret apisix-admin-key -o jsonpath='{.data.G8_APISIX_ADMIN_KEY}' | base64 -d)

kubectl -n g8-gateway exec -i elasticsearch-0 -- sh -c \
  "curl -s -o /dev/null -w '%{http_code}\n' -H 'X-API-KEY: $KEY' \
   -X PUT http://apisix-admin:9180/apisix/admin/global_rules/1 \
   -H 'Content-Type: application/json' --data-binary @-" <<'EOF'
{"plugins":{"prometheus":{"prefer_name":false}}}
EOF

The prometheus plugin must be enabled; plugin_attr is not enough

plugin_attr.prometheus in config.yaml only sets the address the metrics are published on. Unless the plugin is enabled on a global rule, no request metrics are collected at all. The symptom is treacherous: the Prometheus target reads up, /apisix/prometheus/metrics returns 200, and yet a query for apisix_http_status comes back with zero series.

11.6 Accounts, the ingest pipeline and the index template in the log store

Unverified

The commands of this section were run on a development stack on 2026-10-02 (APISIX 3.19.0, Elasticsearch 8.15.3) with the outputs shown. On Elasticsearch 9.5.2 the privileges were measured with roles that name the days by a wildcard, and without the line that reads the template; the roles below name the days by a regular expression, which was measured on 8.15.3 alone. The commands have not been run on this machine yet.

The ingest pipeline, the line of the server's role that reads it and the move of 14.2 were run on 2026-10-04 against a development stack's Elasticsearch 8.15.3 with the outputs shown; that a batch sent twice is stored once was also measured in the lab on Elasticsearch 9.5.2 and OpenSearch 3.9.0.

Two accounts are opened in Elasticsearch, and neither of them is elastic:

account	used by	may
`g8-k3s-gateway`	the gateway, to deliver the request records	write records into the daily indices of this environment; it reads no record, deletes none and sees nothing of the cluster
`g8-k3s-server`	the g8 server, to show the records and to clean them up	read the state of the cluster and g8's ingest pipeline, read the daily indices of this environment and delete a day by its name

The daily indices carry the slug of the environment in g8 (g8-<slug>-access-<yyyy.mm.dd>), so ENV below must be the slug this gateway is registered under in the g8 console. The roles name the days by a regular expression and not by g8-k3s-access-*: a wildcard would also reach the days of an environment whose slug begins with k3s-access.

The passwords of the two accounts are generated into a Secret; they reach neither this document nor the repository:

kubectl -n g8-gateway create secret generic g8-es-accounts \
  --from-literal=gateway="$(openssl rand -base64 24 | tr -d /+= | head -c 24)" \
  --from-literal=server="$(openssl rand -base64 24 | tr -d /+= | head -c 24)"

ENV=k3s
GWP=$(kubectl -n g8-gateway get secret g8-es-accounts -o jsonpath='{.data.gateway}' | base64 -d)
SVP=$(kubectl -n g8-gateway get secret g8-es-accounts -o jsonpath='{.data.server}' | base64 -d)
put() {
  kubectl -n g8-gateway exec -i elasticsearch-0 -- sh -c \
    "curl -s -o /dev/null -w '$1 -> %{http_code}\n' -u elastic:\$ELASTIC_PASSWORD \
     -X PUT http://127.0.0.1:9200$1 -H 'Content-Type: application/json' --data-binary @-"
}
DAYS="/g8-$ENV-access-[0-9]{4}[.][0-9]{2}[.][0-9]{2}/"
put /_security/role/g8-$ENV-gateway <<EOF
{"indices":[{"names":["$DAYS"],"privileges":["index","auto_configure"]}]}
EOF
put /_security/role/g8-$ENV-server <<EOF
{"cluster":["monitor","read_pipeline"],"indices":[{"names":["$DAYS"],
 "privileges":["read","view_index_metadata","monitor","delete_index"]}]}
EOF
put /_security/user/g8-$ENV-gateway <<EOF
{"password":"$GWP","roles":["g8-$ENV-gateway"]}
EOF
put /_security/user/g8-$ENV-server <<EOF
{"password":"$SVP","roles":["g8-$ENV-server"]}
EOF

/_security/role/g8-k3s-gateway -> 200
/_security/role/g8-k3s-server -> 200
/_security/user/g8-k3s-gateway -> 200
/_security/user/g8-k3s-server -> 200

The gateway's account must not be able to read

The password of g8-k3s-gateway is part of the gateway's configuration: whoever can read the global rules of the gateway can obtain it. That is why this account can only add records. When traffic logging is turned on with a plain password, g8 tries the account and refuses one that can read the cluster or a record; a password given as a secret reference is resolved on the gateway and is not tried. create_doc is not enough for the account either: the plugin sends an index action, and with create_doc every record is refused while the request itself answers 200.

The ingest pipeline stores each record under its request id. The gateway's logger sends the records in batches and sends a batch again when the store answers it late, and by then the store has usually written the first one: under the request id, the second copy replaces the first instead of adding a record. The index template below sends every record through this pipeline, so the pipeline is applied first. A store that holds the template and not the pipeline refuses every record while the gateway's bulk request still answers 200; g8 checks that the store holds this version of the pipeline and does not turn traffic logging on without it. The store needs at least one node with the ingest role, which a node carries by default.

kubectl -n g8-gateway exec -i elasticsearch-0 -- sh -c \
  'curl -s -u elastic:$ELASTIC_PASSWORD -X PUT http://127.0.0.1:9200/_ingest/pipeline/g8-access \
   -H "Content-Type: application/json" --data-binary @-' <<'EOF'
{
  "description": "g8 request records: a record is keyed by its request id, so a delivery the gateway sends twice is stored once",
  "version": 1,
  "processors": [
    {
      "set": {
        "field": "_id",
        "value": "{{{request_id}}}",
        "if": "ctx.request_id instanceof String && ctx.request_id.length() == 32"
      }
    }
  ]
}
EOF

{"acknowledged":true}

The index template fixes the type of every field of a request record before the first index of a day is born. The store's administrator applies it once, with the command below. g8 checks that the store holds this version of it and does not turn traffic logging on without it.

kubectl -n g8-gateway exec -i elasticsearch-0 -- sh -c \
  'curl -s -u elastic:$ELASTIC_PASSWORD -X PUT http://127.0.0.1:9200/_index_template/g8-access \
   -H "Content-Type: application/json" --data-binary @-' <<'EOF'
{
  "index_patterns": [
    "g8-*-access-*"
  ],
  "priority": 200,
  "template": {
    "settings": {
      "index": {
        "number_of_shards": 1,
        "codec": "best_compression",
        "auto_expand_replicas": "0-1",
        "default_pipeline": "g8-access",
        "analysis": {
          "analyzer": {
            "g8_nodes": {
              "type": "pattern",
              "pattern": "(\\s*,\\s*)|(\\s+:\\s+)",
              "lowercase": false
            }
          }
        }
      }
    },
    "mappings": {
      "dynamic": false,
      "properties": {
        "start_time": {
          "type": "date",
          "format": "epoch_millis"
        },
        "latency": {
          "type": "float"
        },
        "upstream_latency": {
          "type": "float"
        },
        "apisix_latency": {
          "type": "float"
        },
        "client_ip": {
          "type": "ip",
          "ignore_malformed": true
        },
        "route_id": {
          "type": "keyword"
        },
        "service_id": {
          "type": "keyword"
        },
        "upstream": {
          "type": "keyword",
          "ignore_above": 1024,
          "fields": {
            "node": {
              "type": "text",
              "analyzer": "g8_nodes",
              "norms": false,
              "index_options": "docs"
            }
          }
        },
        "consumer": {
          "properties": {
            "username": {
              "type": "keyword"
            }
          }
        },
        "request": {
          "properties": {
            "method": {
              "type": "keyword"
            },
            "uri": {
              "type": "wildcard"
            },
            "url": {
              "type": "keyword",
              "index": false,
              "doc_values": false
            },
            "size": {
              "type": "long"
            },
            "headers": {
              "type": "object",
              "enabled": false
            },
            "querystring": {
              "type": "object",
              "enabled": false
            }
          }
        },
        "request_body": {
          "type": "text",
          "index": false
        },
        "response": {
          "properties": {
            "status": {
              "type": "short"
            },
            "size": {
              "type": "long"
            },
            "headers": {
              "type": "object",
              "enabled": false
            },
            "body": {
              "type": "text",
              "index": false
            }
          }
        },
        "server": {
          "properties": {
            "hostname": {
              "type": "keyword"
            },
            "version": {
              "type": "keyword"
            }
          }
        },
        "request_id": {
          "type": "keyword"
        },
        "route_name": {
          "type": "keyword"
        },
        "service_name": {
          "type": "keyword"
        },
        "upstream_status": {
          "type": "keyword"
        },
        "x_request_id": {
          "type": "keyword"
        },
        "logged_at": {
          "type": "date",
          "format": "epoch_second"
        }
      },
      "_meta": {
        "g8_records": 2
      },
      "_source": {
        "excludes": [
          "response.headers.set-cookie"
        ]
      }
    }
  }
}
EOF

{"acknowledged":true}

What the two accounts may and may not do, asked of the store itself. The day the check writes is deleted by its last line:

GWP=$(kubectl -n g8-gateway get secret g8-es-accounts -o jsonpath='{.data.gateway}' | base64 -d)
SVP=$(kubectl -n g8-gateway get secret g8-es-accounts -o jsonpath='{.data.server}' | base64 -d)
kubectl -n g8-gateway exec -i elasticsearch-0 -- sh -s "$GWP" "$SVP" <<'EOF'
curl -s -o /dev/null -w 'g8-k3s-gateway reads the cluster      -> %{http_code}\n' -u "g8-k3s-gateway:$1" http://127.0.0.1:9200/
curl -s -o /dev/null -w 'g8-k3s-gateway reads records          -> %{http_code}\n' -u "g8-k3s-gateway:$1" 'http://127.0.0.1:9200/g8-k3s-access-*/_search'
curl -s -o /dev/null -w 'g8-k3s-gateway writes a day           -> %{http_code}\n' -u "g8-k3s-gateway:$1" -X POST http://127.0.0.1:9200/g8-k3s-access-2000.01.01/_doc -H 'Content-Type: application/json' -d '{}'
curl -s -o /dev/null -w 'g8-k3s-gateway writes beside its days -> %{http_code}\n' -u "g8-k3s-gateway:$1" -X POST http://127.0.0.1:9200/g8-k3s-access-x-access-2000.01.01/_doc -H 'Content-Type: application/json' -d '{}'
curl -s -o /dev/null -w 'g8-k3s-server reads the template      -> %{http_code}\n' -u "g8-k3s-server:$2" http://127.0.0.1:9200/_index_template/g8-access
curl -s -o /dev/null -w 'g8-k3s-server reads the pipeline      -> %{http_code}\n' -u "g8-k3s-server:$2" http://127.0.0.1:9200/_ingest/pipeline/g8-access
curl -s -o /dev/null -w 'g8-k3s-server reads beside its days   -> %{http_code}\n' -u "g8-k3s-server:$2" 'http://127.0.0.1:9200/g8-k3s-access-x-access-2000.01.01/_search'
curl -s -o /dev/null -w 'g8-k3s-server writes a record         -> %{http_code}\n' -u "g8-k3s-server:$2" -X POST http://127.0.0.1:9200/g8-k3s-access-2000.01.01/_doc -H 'Content-Type: application/json' -d '{}'
curl -s -o /dev/null -w 'g8-k3s-server deletes the day         -> %{http_code}\n' -u "g8-k3s-server:$2" -X DELETE http://127.0.0.1:9200/g8-k3s-access-2000.01.01
EOF

g8-k3s-gateway reads the cluster      -> 403
g8-k3s-gateway reads records          -> 403
g8-k3s-gateway writes a day           -> 201
g8-k3s-gateway writes beside its days -> 403
g8-k3s-server reads the template      -> 200
g8-k3s-server reads the pipeline      -> 200
g8-k3s-server reads beside its days   -> 403
g8-k3s-server writes a record         -> 403
g8-k3s-server deletes the day         -> 200

Data streams cannot be used

elasticsearch-logger writes an index action in its bulk request; data streams require create and reject index. Hence one dated index per day.

11.7 Turning on the request records

Unverified

These steps were walked on a development stack on 2026-10-02, with the two accounts of 11.6. They have not been walked against this machine yet.

The request records are turned on from the g8 console, once this gateway is registered there as the environment k3s. No command on this machine writes the logger's rule.

Environments › k3s › Edit › Observability. Elasticsearch URL: the address the store is exposed at in 11.3; the NetworkPolicy of this namespace admits the g8 server through that address and no other. Username g8-k3s-server, password the server key of the g8-es-accounts Secret. Test connection, then save.
Observe › Traffic Explorer › Settings. The Status card must read Log store Connected, Index template Applied and Ingest pipeline Applied.
Turn on traffic logging. Log store address, as the gateway reaches it: http://elasticsearch:9200. Username g8-k3s-gateway, password the gateway key of the g8-es-accounts Secret. Turn on.

g8 writes one global rule, g8-request-records, beside rule 1: the logger with the daily index of this environment, the fields the records need and the default masks. What is recorded and masked is changed afterwards on the same Settings page. Where the gateway writes is shown on the Status card as Gateway writes to; Change address gives the logger another address and account for the same store, through the same checks of the account as turning on.

11.8 Retention

A log store does not shrink on its own. With no limit the disk fills eventually, and past 95% Elasticsearch turns its indices read-only.

g8 deletes the request records: once a day it deletes, by its name, every daily index older than the number of days the environment keeps — 30 unless it is changed under Traffic Explorer › Settings › Retention › Change — and writes each deletion to the audit log. The store carries no lifecycle policy for these indices; a second deleter would delete days g8 still shows.

A day is named by its UTC date, by the gateway that writes it and by g8 that reads and deletes it.

On a single-node Elasticsearch a replica can never be allocated. The template sets auto_expand_replicas: 0-1: an index has no replica on one node and one as soon as a second node joins. The verification below must show 0 and green.

Prometheus is given two limits at once: retention.time=40d and retention.size=5GB. Whichever fills first does the cutting. Forty days keep the whole 30-day window of Analytics.

11.9 Verification

Unverified

The output below was recorded on this machine while the rule of the earlier guide was in place; the index it shows is named by the machine's local day. With the rule g8 writes, the commands were run on a development stack on 2026-10-02: the index of the UTC day showed the requests sent, 0 and green. They have not been run on this machine since.

kubectl -n g8-gateway exec elasticsearch-0 -- sh -c 'curl -s -u elastic:$ELASTIC_PASSWORD \
  "http://127.0.0.1:9200/_cat/indices/g8-*?v&h=index,docs.count,rep,health"'

kubectl -n g8-gateway exec elasticsearch-0 -- sh -c \
  'curl -s "http://prometheus:9090/api/v1/targets?state=active"'

index                    docs.count rep health
g8-k3s-access-2026.08.29         30   0 green

Every APISIX pod must appear as up in the Prometheus targets.

Traffic Explorer in the g8 console must list the same requests.

Analytics reads two things from these metrics: each route under its id with its name beside it, and each bucket counted exactly with the anchored range selector:

kubectl -n g8-gateway exec elasticsearch-0 -- sh -c \
  'curl -sG http://prometheus:9090/api/v1/query --data-urlencode "query=count by (route, route_name) (apisix_http_status)"'
kubectl -n g8-gateway exec elasticsearch-0 -- sh -c \
  'curl -sG http://prometheus:9090/api/v1/query --data-urlencode "query=sum(increase(apisix_http_status[5m] anchored))"'

The first query lists every route that answered with its id in route and its name in route_name; a route that carries a name means a copy of the prometheus plugin has prefer_name on, and Analytics reads that route empty. The second must answer "status":"success"; a Prometheus started without --enable-feature=promql-extended-range-selectors answers anchored modifier is experimental and not enabled, and Analytics marks its counts as estimates. Both queries were run on this machine on 2026-10-04 against Prometheus 3.15.0: every route with a name showed it in route_name, and the anchored query answered success.

Request records are LOST while Elasticsearch is unreachable

The gateway sends a batch of records once more after the first failure (max_retry_count 1 in the rule g8 writes) and then drops it. The loss was measured with the rule of the earlier guide, which tried three times: on a production installation during a password change, 20 requests that arrived while Elasticsearch was restarting appeared in the metrics but never reached the index.

Read the loss from APISIX's error log, not from a counter difference:

grep -c 'elasticsearch-logger' <error.log>

Subtracting the document count from the metric counter is not reliable and produces false alarms. A measured example: counter 88, documents 43, difference 45 — with not one delivery error in the log. The difference has three causes and none of them is loss: the counter counts from the moment APISIX started (requests from before the plugin was enabled are in the counter but not the index), a restart resets the counter while the index persists, and the two sides have different retention. The difference is a hint only within a single window — no APISIX restart, nothing deleted by retention, both plugins enabled together.

An installation that cannot afford losing records must put a buffer in between (kafka-logger, or a local file-logger plus a separate shipper).

Deleting pods was measured to lose nothing: after elasticsearch-0 and the Prometheus pod were deleted and came back, the document count was unchanged. Persistence comes from the PVCs; delete a PVC and the data is gone.

12. Diagnostics

You will see error lines on a healthy installation too

logs output is cumulative: it holds everything since the container or pod was born. APISIX prints connection refused while waiting for etcd at startup, and the same lines land again for members stopped during the quorum drill. Those lines describe the past, not the present state.

Measured on a healthy installation in this environment:

request etcd endpoint 'http://etcd-1:2379/version' error, connection refused
error.log lines: 0

An empty error.log is normal too — APISIX's own error log is a separate file, and the startup messages go to stdout rather than there.

The present state is what the verification section's counts tell you; logs is for diagnosis, not proof of health. Decide whether a line is current from its timestamp.

kubectl -n g8-gateway get pod -o wide
kubectl -n g8-gateway logs -l app=apisix --tail=50 --prefix
kubectl -n g8-gateway logs etcd-0 --tail=50
kubectl -n g8-gateway describe pod etcd-0
kubectl -n g8-gateway get events --sort-by=.lastTimestamp | tail -20

kubectl logs reads only the current log file, not the rotated ones. kubelet keeps 10Mi × 5 per container, but the window kubectl logs sees is just that 10Mi. The access-log line measured on this cluster is 141 bytes (short URI, short User-Agent); 10Mi holds about 74,000 of them. Real traffic has longer URIs and User-Agents, so lines grow and the window shrinks — at a hundred requests a second spread over three pods, each pod keeps roughly half an hour. For anything older, read the rotated files on the node under /var/log/pods/<namespace>_<pod>_*/ — or ship the logs off the cluster.

Prometheus is tested separately on each pod:

for p in $(kubectl -n g8-gateway get pod -l app=apisix -o jsonpath='{.items[*].metadata.name}'); do
  kubectl -n g8-gateway port-forward pod/$p 19091:9091 >/dev/null 2>&1 &
  PF=$!
  sleep 4
  echo "$(curl -s -o /dev/null -w '%{http_code}' http://127.0.0.1:19091/apisix/prometheus/metrics) $(curl -s http://127.0.0.1:19091/apisix/prometheus/metrics | grep -c '^apisix_')"
  kill $PF 2>/dev/null
  sleep 1
done

87
87
87

13. Uninstalling

Delete the environment in the g8 console first, or clear its log store there: g8 connects to the store once a day to delete the days past the retention, and records a failed cleanup every day for as long as the environment points at a store that is gone.

kubectl delete namespace g8-gateway

namespace "g8-gateway" deleted

Deleting the namespace deletes the PVCs too; because local-path uses the Delete reclaim policy, the etcd data and the backups are gone permanently.

kubectl get ns g8-gateway --ignore-not-found | wc -l
kubectl get pv --no-headers | grep -c g8-gateway || true

0
0

14. Upgrading

Unverified

A version upgrade was not attempted in this environment. The sequence is the same as in the OKD document (04-okd, section 14): backup first, then APISIX one pod at a time, and etcd last without skipping versions.

On 2026-09-30 the APISIX Deployment moved from 3.18.0 to 3.19.0 with kubectl set image, rolling, and the data plane answered Server: APISIX/3.19.0; etcd was not touched. Whether the backup and the per-step verification ran for that change is not recorded, so the sequence stays unverified.

14.1 Request records of an installation made with the earlier guide

Unverified

This move was rehearsed on a development stack on 2026-10-02: an installation made as the earlier guide wrote it, then the steps below in this order, with the outputs shown. It has not been carried out on this machine yet.

The earlier version of this guide wrote the logger into global rule 1 by hand: with the elastic account, an index named g8-k3s-access- and the machine's local day, and a lifecycle policy that deleted the indices. Such an installation is moved in this order:

Do not turn traffic logging on before step 6

Settings offers Turn on traffic logging as soon as the pipeline and the template are applied. Turning on over the old rule completes it in place: the account is neither asked for nor tried, and elastic stays in the rule.

Open the two accounts (11.6).
Connect the store in the g8 console with g8-k3s-server (step 1 of 11.7). Settings then reads Index template Outdated, Ingest pipeline Missing and the logger rule Customized. From its next daily cleanup on, g8 deletes the days past the retention.
Apply the ingest pipeline and then the index template (11.6). They apply to the days born after them.
Take the lifecycle policy off the indices and delete it:

kubectl -n g8-gateway exec elasticsearch-0 -- sh -c 'curl -s -u elastic:$ELASTIC_PASSWORD \
  -X PUT "http://127.0.0.1:9200/g8-*-access-*/_settings" \
  -H "Content-Type: application/json" -d "{\"index.lifecycle.name\":null}"'
kubectl -n g8-gateway exec elasticsearch-0 -- sh -c 'curl -s -u elastic:$ELASTIC_PASSWORD \
  -X DELETE http://127.0.0.1:9200/_ilm/policy/g8-access-retention'

{"acknowledged":true}{"acknowledged":true}

The policy kept the indices 14 days. g8 keeps them 30 days unless that is changed: set the number of days this store is sized for under Traffic Explorer › Settings › Retention › Change.
Run the command of 11.5 again. It prints 200 here; rule 1 then carries the metrics alone and Settings reads the logger rule Off. Only then turn traffic logging on, with g8-k3s-gateway (steps 2 and 3 of 11.7).
Change the password of elastic: it was part of the gateway's configuration for as long as the old rule existed.

No record is written between the two halves of step 6. Every day written before the move, and the day of the move itself, was born without g8's template and without the fields its records carry: Traffic Explorer cannot list those days, and Settings counts them as written under another version. They stay in the store until the retention deletes them. The records are listed from the first day whose index is born after the move.

g8 lists and deletes the days of the environment's own slug alone. If this gateway is registered in g8 under another slug than k3s, the days the old rule wrote have no deleter once the policy is gone: delete them by their names when they are no longer needed.

14.2 The ingest pipeline on an installation made before it

Unverified

This move was rehearsed on 2026-10-04 against a development stack's Elasticsearch 8.15.3: an installation with the earlier template and a day born under it, then the steps below in this order, with the outputs shown. It has not been carried out on this machine yet.

An installation made before this version of the guide holds the index template without the ingest pipeline, and its server account may not read pipelines. Settings then reads Index template Outdated and Ingest pipeline Cannot be read. The gateway keeps writing records meanwhile, copies included, and the settings can still be saved; g8 refuses only to turn logging on anew until both read Applied. The move, in this order:

Run the block that writes the two roles and the two users again (11.6). It keeps the passwords and gives the server's role the line that reads the pipeline; every line answers 200.
Apply the ingest pipeline (11.6).
Apply the index template (11.6). The days born after it send their records through the pipeline.
Send the day of today, born under the earlier template, through the pipeline at once. On a store that serves several environments, run it once for each environment's slug:

DAY=$(date -u +%Y.%m.%d)
kubectl -n g8-gateway exec elasticsearch-0 -- sh -c \
  "curl -s -u elastic:\$ELASTIC_PASSWORD -X PUT http://127.0.0.1:9200/g8-k3s-access-$DAY/_settings \
   -H 'Content-Type: application/json' -d '{\"index.default_pipeline\":\"g8-access\"}'"

{"acknowledged":true}

A store that holds no record of today yet answers 404 index_not_found_exception here: today's day will be born under the new template, and nothing more is needed. Settings then reads both Applied. A day written before the move may hold copies of a batch the gateway delivered twice; it stays as it is until the retention deletes it.

15. Traps hit during installation

15.1 `runAsNonRoot` blocks the pod when the image's user is a name

Symptom: the pod stays in CreateContainerConfigError. The image is pulled, the pod is scheduled, but the container is never created.

container has runAsNonRoot and image has non-numeric user (apisix), cannot verify user is non-root

The APISIX image declares USER apisix — a name, not a number. kubelet does not read the image's /etc/passwd to translate it, and since it cannot verify the user is not root it refuses. The fix is to write the numeric uid explicitly:

      securityContext:
        runAsNonRoot: true
        runAsUser: 636
        runAsGroup: 636

This trap is invisible on OKD: restricted-v2 already injects a numeric runAsUser. It is the concrete example of the rule that what is green on k3s must be tested on OKD, and the reverse.

15.2 With `OrderedReady`, the etcd bootstrap deadlocks

Symptom: three replicas are requested but only one pod is created, and it stays at 0/1.

etcd-0 0/1 Running

The default podManagementPolicy is OrderedReady: the StatefulSet does not create etcd-1 until etcd-0 is Ready. etcd-0's readiness probe is /readyz, and in a three-member cluster it can never be ready alone because it cannot form a quorum.

The field is immutable; the StatefulSet is deleted and recreated, and the PVCs left over from the failed bootstrap are deleted too.

15.3 Hairpin SNAT exists on k3s as well

Symptom: there is no direct symptom — if allow_admin is wide enough it is never noticed. It is real where it is not noticed, too.

The source addresses of eight in-cluster requests to the apisix-admin Service:

4  10.42.0.27 -> 200
4  10.42.0.1  -> 200

When a request is routed back to its own pod, the source address is translated to the cni0 bridge gateway (10.42.0.1). That address is inside the pod network, so an allow_admin containing 10.42.0.0/16 already covers it.

The same mechanism exists on OKD, but there the SNAT address is link-local (169.254.0.5) and falls outside the pod network; unless it is added to allow_admin by hand, requests get random 403s (04-okd, 14.3). The same product, the same configuration, two different outcomes.

The real lesson: in Kubernetes the source IP is not access control; the control is the NetworkPolicy in section 8.

15.4 The etcd image contains no shell

Symptom:

executable file `sh` not found in $PATH: No such file or directory

The etcd image carries only the etcd binaries. kubectl exec … -- sh -c does not work; commands are given directly. Shell substitutions such as $(date) cannot be used in the manifest either — Kubernetes' own $(VAR) substitution is used. Steps that do need a shell (pruning files) run as an initContainer from an image that has one.

15.5 `kubectl logs -l` is not chronological

Symptom: eight requests are made, the last eight lines are taken from the log, and the numbers do not add up — all eight returned 200 while the list shows 401 and 201.

10.42.0.27 -> 200
127.0.0.1  -> 201
127.0.0.1  -> 401

With -l, the logs of several pods are appended one after another, not merged by time. tail therefore returns old lines from a different pod. The JSON line starts with the time field, so lexical order is chronological order; a sort goes in between.

15.6 A liveness probe on cluster health kills the last member

Symptom: during quorum loss the one surviving etcd member is restarted over and over. kubectl describe shows:

Killing … (x6 over 10m) Container etcd failed liveness probe, will be restarted

etcdctl endpoint health is a cluster health check: it performs a read that requires quorum. Put the same command on both readiness and liveness and, on quorum loss, readiness failing is correct (the pod should leave the Service) while liveness failing is a disaster — kubelet kills the last pod holding the data roughly every 90 seconds, and the loop does not stop until quorum returns.

Since 3.5.12 etcd exposes two endpoints precisely to separate the two. Measured:

no quorum:
  /livez  -> HTTP/1.1 200 OK
  /readyz -> HTTP/1.1 503 Service Unavailable

That is why the manifest in section 5 points liveness at /livez and readiness at /readyz. Before and after the fix, across three minutes of quorum loss:

etcdctl endpoint health  ->  restartCount 1 → 5 → 6
/livez + /readyz         ->  restartCount 0 → 0 → 0

The rule: liveness must ask "is the process running", not "is the cluster healthy". The second question belongs to readiness.

15.7 The `NOSPACE` alarm is a one-way door

Symptom: writes are refused with etcdserver: mvcc: database space exceeded. Nobody notices, because the data plane keeps returning 200 — and once it is noticed, freeing space is not enough.

The recovery chain was measured, and all three steps are required:

159 writes         -> dbSize=16789504  alarm=NOSPACE
delete + compact   -> dbSize=16789504  write refused
defrag             -> dbSize=122880    write refused
alarm disarm       -> write OK

There are three separate misconceptions here, each verified individually:

Deleting keys does not free space. etcd keeps the revision history; a delete writes a new revision.
Compaction does not return disk space. dbSizeInUse drops, dbSize does not move at all — the freed pages stay free inside etcd and never return to the filesystem. Since the quota watches dbSize, the quota is still full. The extreme measured in this environment: dbSizeInUse at 41 kB while dbSize sat at 98 MB.
Defrag returns the space but does not lower the alarm. The alarm is sticky: it survives a process restart, a container recreation and a host reboot.

Recovery:

for n in 0 1 2; do
  kubectl -n g8-gateway exec etcd-$n -- etcdctl --command-timeout=900s defrag
done
kubectl -n g8-gateway exec etcd-0 -- etcdctl --endpoints=http://etcd-0.etcd:2379,http://etcd-1.etcd:2379,http://etcd-2.etcd:2379 alarm disarm

Leave alarm disarm for last: disarming without actually freeing space puts you straight back where you were on the next write.

The lasting fix is scheduled defragmentation — see long-term operation. With no defrag scheduled, dbSize keeps growing even when compaction is working correctly, and the quota fills sooner or later.

15.8 A member that loses its disk cannot rejoin on its own

Symptom: a member loses its data directory (disk failure, accidental deletion, a lost PVC). When it comes back it loops forever, logging cluster ID mismatch or member has already been bootstrapped.

What makes it insidious: member list still shows that member as started and the data plane keeps returning 200. The cluster silently drops to 2/3, and losing one more member takes quorum with it.

The cause is ETCD_INITIAL_CLUSTER_STATE: new hardcoded in the manifest. Seeing an empty data directory, etcd tries to bootstrap a new cluster rather than join the existing one.

The recovery chain was measured:

STATE=new + bos veri dizini    -> cluster ID mismatch (uye asla katilmaz)
member remove + olcek dusur    -> veri dizini gercekten bos
member add (saglikli kume)     -> ETCD_INITIAL_CLUSTER=...
STATE=existing, tek acilis     -> healthy=3/3
STATE=new'e donus              -> healthy=3/3, veri yerinde

The critical detail: STATE is only read while the data directory is empty. So existing is needed for exactly one start; once the member has joined it runs indefinitely under its normal definition.

Two steps are the easiest to miss and both are mandatory:

Scale down. Deleting the pod is not enough; the StatefulSet recreates it immediately and etcd rebuilds the data directory. The member has to be gone before you wipe the directory.
Verify the remaining cluster is healthy. member add refuses with etcdserver: unhealthy cluster while the cluster is unhealthy.

When wiping the data directory, delete hidden entries too; the pattern rm -rf dir/* skips entries beginning with ..

Recovery:

E=http://etcd-0.etcd:2379,http://etcd-1.etcd:2379,http://etcd-2.etcd:2379
E2=http://etcd-0.etcd:2379,http://etcd-1.etcd:2379

ID=$(kubectl -n g8-gateway exec etcd-0 -- etcdctl --endpoints=$E member list -w simple | grep 'etcd-2' | cut -d, -f1)
kubectl -n g8-gateway exec etcd-0 -- etcdctl --endpoints=$E2 member remove "$ID"
kubectl -n g8-gateway scale sts etcd --replicas=2
kubectl -n g8-gateway delete pvc data-etcd-2

kubectl -n g8-gateway exec etcd-0 -- etcdctl --endpoints=$E2 endpoint health
ADD=$(kubectl -n g8-gateway exec etcd-0 -- etcdctl --endpoints=$E2 \
  member add etcd-2 --peer-urls=http://etcd-2.etcd:2380)
IC=$(echo "$ADD" | grep -oE 'ETCD_INITIAL_CLUSTER="[^"]*"' | sed 's/ETCD_INITIAL_CLUSTER="//; s/"$//')

kubectl -n g8-gateway set env sts/etcd ETCD_INITIAL_CLUSTER_STATE=existing "ETCD_INITIAL_CLUSTER=$IC"
kubectl -n g8-gateway scale sts etcd --replicas=3

Once the member has joined, return to the normal definition:

kubectl -n g8-gateway set env sts/etcd ETCD_INITIAL_CLUSTER_STATE=new \
  "ETCD_INITIAL_CLUSTER=etcd-0=http://etcd-0.etcd:2380,etcd-1=http://etcd-1.etcd:2380,etcd-2=http://etcd-2.etcd:2380"

15.9 ConfigMap and Secret changes never reach the pods

Symptom: you changed config.yaml, the ConfigMap was updated, and nothing changed. There is no error; the pods keep reading the old content.

The ConfigMap is mounted with subPath — a single file rather than the whole directory. Kubernetes does not update files mounted via subPath; this is documented behaviour, not a defect. The same applies to the Secret holding the admin key.

Measured:

ConfigMap d04c459e -> pod d04c459e
ConfigMap 6c4bb814 -> 90 s sonra pod hala d04c459e
rollout restart    -> pod 6c4bb814

The way to put a change into effect is to recreate the pods:

kubectl -n g8-gateway rollout restart deploy/apisix
kubectl -n g8-gateway rollout status deploy/apisix --timeout=300s

This is also how the admin key is rotated: update the Secret, then rollout restart. The restart is not disorderly — the Deployment replaces pods one at a time, and the PodDisruptionBudget in 7.1 guarantees at least two pods stay up.

16. Differences from OKD

Every row was measured; looking at one and assuming the other is precisely what this table exists to prevent.

topic	OKD	k3s
Security context	Injected by `restricted-v2`	Written by hand in the manifest
Runtime uid	1000770000 (namespace range)	636 (APISIX) / 1000 (etcd), fixed
`runAsNonRoot` alone	Enough (the SCC supplies a numeric uid)	Not enough — the pod will not start (15.1)
Storage class	`topolvm-provisioner`	`local-path`, `WaitForFirstConsumer`
External access	Route	Ingress + Traefik
Source IP (external request)	`10.128.0.2` (OVN gateway)	`10.42.0.8` (Traefik pod IP)
Hairpin SNAT address	`169.254.0.5` — outside the pod network	`10.42.0.1` — inside the pod network
Extra `allow_admin` entry	`169.254.0.0/17` required	Not required
Blocked by NetworkPolicy	Silent drop (timeout)	`Connection refused`
Container log rotation	50Mi × 5 = 250 MB	10Mi × 5 = 50 MB
`ulimit -n`	1048576 (CRI-O)	1048576 (containerd)
Admin API on quorum loss	First request ~42 s, later ones ~0.02 s	Identical

17. Long-term operation

topic	setting	why
etcd growth	`ETCD_AUTO_COMPACTION_MODE=periodic`, `RETENTION=1`	Without compaction etcd keeps the old revision of every write
etcd quota	`ETCD_QUOTA_BACKEND_BYTES=2147483648`	The PVC is 2Gi; the quota must be smaller than the disk so etcd alarms before it fills
Logs	`access_log: /dev/stdout`, `error_log: /dev/stderr`	Writing to the container's filesystem is not durable and is not rotated
Log rotation	kubelet: 10Mi × 5	50 MB per container. A fifth of OKD's; under heavy traffic the history stays short
File descriptors	no setting needed	The containerd default measured 1048576
Placement	`podAntiAffinity: preferred`	On a single node `required` leaves replicas unschedulable. Make it `required` on multiple nodes
Storage	`local-path` is node-bound	The PVCs live on that node's disk; if the node goes, the data goes. A multi-node cluster needs a network-backed class
Backup	Daily CronJob, 14 files	Section 9
etcd access	NetworkPolicy	etcd has no authentication

18. Scope limits and next steps

What this installation does not do, stated plainly:

It was verified on a single-node cluster. Because local-path is node-bound, a multi-node cluster must also change the storage class; that was not tried.
There is no TLS. Traffic between APISIX and etcd, the Admin API and the Ingress are all plain HTTP.
The real client IP does not reach APISIX — 10.5.
etcd has no authentication. Access is restricted only by NetworkPolicy.
Restore and upgrade were not attempted.
Behaviour under load was not measured.
There is no HorizontalPodAutoscaler or resource quota. A PodDisruptionBudget is created in 7.1.

G8 dev · Apache APISIX® 3.19.0

Apache, Apache APISIX and APISIX are trademarks of The Apache Software Foundation.

1. What we are installing​

2. Prerequisites​

2.1 Ports​

3. Namespace and image pull​

4. Security context​

5. The etcd cluster​

6. Admin key​

7. APISIX and Ingress​

7.1 PodDisruptionBudget​

8. NetworkPolicy​

9. Backup​

9.1 Defragmentation​

10. Verification​

10.1 Pods and the security context​

10.2 The etcd cluster​

10.3 Admin API and key substitution​

10.4 Does the configuration propagate to all three pods​

10.5 The Ingress and the source IP​

10.6 NetworkPolicy​

10.7 Quorum: one member down​

10.8 Quorum loss​

10.9 Is the backup valid​

10.10 Cleaning up verification leftovers​

11. Observability​

11.1 Elasticsearch​

11.2 Prometheus and NetworkPolicy​

11.3 Exposing​

11.4 Install​

11.5 Wiring APISIX​

11.6 Accounts, the ingest pipeline and the index template in the log store​

11.7 Turning on the request records​

11.8 Retention​

11.9 Verification​

12. Diagnostics​

13. Uninstalling​

14. Upgrading​

14.1 Request records of an installation made with the earlier guide​

14.2 The ingest pipeline on an installation made before it​

15. Traps hit during installation​

15.1 runAsNonRoot blocks the pod when the image's user is a name​

15.2 With OrderedReady, the etcd bootstrap deadlocks​

15.3 Hairpin SNAT exists on k3s as well​

15.4 The etcd image contains no shell​

15.5 kubectl logs -l is not chronological​

15.6 A liveness probe on cluster health kills the last member​

15.7 The NOSPACE alarm is a one-way door​

15.8 A member that loses its disk cannot rejoin on its own​

15.9 ConfigMap and Secret changes never reach the pods​

16. Differences from OKD​

17. Long-term operation​

18. Scope limits and next steps​

1. What we are installing

2. Prerequisites

2.1 Ports

3. Namespace and image pull

4. Security context

5. The etcd cluster

6. Admin key

7. APISIX and Ingress

7.1 PodDisruptionBudget

8. NetworkPolicy

9. Backup

9.1 Defragmentation

10. Verification

10.1 Pods and the security context

10.2 The etcd cluster

10.3 Admin API and key substitution

10.4 Does the configuration propagate to all three pods

10.5 The Ingress and the source IP

10.6 NetworkPolicy

10.7 Quorum: one member down

10.8 Quorum loss

10.9 Is the backup valid

10.10 Cleaning up verification leftovers

11. Observability

11.1 Elasticsearch

11.2 Prometheus and NetworkPolicy

11.3 Exposing

11.4 Install

11.5 Wiring APISIX

11.6 Accounts, the ingest pipeline and the index template in the log store

11.7 Turning on the request records

11.8 Retention

11.9 Verification

12. Diagnostics

13. Uninstalling

14. Upgrading

14.1 Request records of an installation made with the earlier guide

14.2 The ingest pipeline on an installation made before it

15. Traps hit during installation

15.1 `runAsNonRoot` blocks the pod when the image's user is a name

15.2 With `OrderedReady`, the etcd bootstrap deadlocks

15.3 Hairpin SNAT exists on k3s as well

15.4 The etcd image contains no shell

15.5 `kubectl logs -l` is not chronological

15.6 A liveness probe on cluster health kills the last member

15.7 The `NOSPACE` alarm is a one-way door

15.8 A member that loses its disk cannot rejoin on its own

15.9 ConfigMap and Secret changes never reach the pods

16. Differences from OKD

17. Long-term operation

18. Scope limits and next steps