Skip to main content

Rotate the secrets key

The key in G8_SECRETS_KEY seals every secret G8 stores: the admin key of each environment, the password of its log store, the bind password of the directory connection, the mail relay's password and the signing key of each webhook channel. Changing that key is a three-phase procedure. The server does the resealing itself at start; your part is to give it the old and the new key in the right order and to roll the pods between the phases.

Who may do it​

Whoever runs the server and may change its settings, which on OKD means the g8-secrets Secret. Nothing in the console changes the key, and no role in the console is needed.

How the server uses the key​

A sealed value carries a short id of the key that sealed it. At start the server builds a sealer from the current key and, when G8_SECRETS_KEY_PREVIOUS is set, from the previous one as well. It then reads every sealed row, opens it with whichever of the two keys the row names, and writes back any row that the previous key sealed, now sealed under the current one. The log line sealed secrets moved to the current key reports the number of rows it rewrote, under rows.

A row sealed under a key the process holds neither as current nor as previous stops the start, with the error sealed with a key this server does not hold and the table and the row named. The readiness address /readyz runs the same check on every call: it answers 200 only when the process can open every sealed row, and 503 otherwise. A pod answering 503 leaves the Service, so it serves no request it could not complete.

The Deployment the chart installs replaces its pods with the strategy Recreate: the running pods stop before the new ones start. That matters here, because the reseal rewrites rows that every replica shares. A pod of the old generation that kept running would hold only the old key and could no longer open the rows the new generation moved. Two key generations never serve at once.

Before you start​

  • A new key: 64 hexadecimal characters, which is 32 random bytes. The chart's install notes give the command.

    openssl rand -hex 32

  • The current key, saved where your team keeps secrets. Rolling back needs it, and a backup taken before today needs it; see Back up and restore.

  • A fresh dump of the database, as the same page describes.

  • A moment when the console may be unavailable: each roll stops every pod before it starts the new ones.

Phase 1: start with both keys​

  1. Put the current key into G8_SECRETS_KEY_PREVIOUS and the new key into G8_SECRETS_KEY. On OKD both are keys of the g8-secrets Secret, which the install notes list. One way to rewrite the Secret is below; your cluster team may have another.

    oc create secret generic g8-secrets -n <namespace> \
      --from-literal=G8_SECRETS_KEY=<new key> \
      --from-literal=G8_SECRETS_KEY_PREVIOUS=<current key> \
      --dry-run=client -o yaml | oc apply -f -

  2. Roll the Deployment, so every pod starts again and reads the Secret.

    oc rollout restart deployment/g8 -n <namespace>
    oc rollout status deployment/g8 -n <namespace>

  3. Read the log of a new pod. The line sealed secrets moved to the current key carries the number of rows moved under rows. The line is absent when no row needed moving, which happens when the server was started again before the Secret changed.

Run in the lab on an installation with one environment: the new pod printed the line with one row moved, the environment's sealed admin key.

"msg":"schema ready","version_from":16,"version_to":16,"applied":null
"msg":"sealed secrets moved to the current key","rows":1
"msg":"listening","addr":"[::]:8080","tls":false

Phase 2: check readiness​

  1. Call /readyz on each new pod, or on the Service once the rollout is complete.

    curl -sS -o /dev/null -w '%{http_code}\n' https://<console host>/readyz

  2. Read the answer. 200 with the status ready means the process opened every sealed row with the keys it holds. 503 with the status sealed secret needs a key this instance does not hold means a pod started without one of the two keys. That pod's log carries the error not ready: a sealed row needs a key this instance does not hold with the row named.

  3. Sign in to the console and open an environment. Its cluster still answers, which shows that its admin key opened under the new key.

Run in the lab: /readyz answered 200 through the Route after phase 1 and again after phase 3.

Phase 3: remove the previous key​

  1. Rewrite the g8-secrets Secret with G8_SECRETS_KEY alone, holding the new key, and no G8_SECRETS_KEY_PREVIOUS. The Deployment reads the previous key as optional, so a Secret without it is complete.

    oc create secret generic g8-secrets -n <namespace> \
      --from-literal=G8_SECRETS_KEY=<new key> \
      --dry-run=client -o yaml | oc apply -f -

  2. Roll the Deployment again, as in phase 1.

  3. Check /readyz again, as in phase 2. From now on the old key opens nothing in this database. Keep it only for dumps taken before the rotation.

Run in the lab: after the roll with the new key alone, the pod's log carried no moved-rows line and /readyz answered 200. The lab then went one step further on purpose and gave the Secret a key the rows were not sealed under. The pod never became ready and the Route answered 503 until the right key was put back; its log ended with the error below.

"level":"ERROR","msg":"server exited","error":"environments.engine_admin_key for id=docs: sealed with a key this server does not hold"

Rolling back after phase 1​

After phase 1 the rows are sealed under the new key. A pod carrying only the old key stops at start with the table and row named. Going back therefore needs both keys, swapped: put the new key into G8_SECRETS_KEY_PREVIOUS and the old key into G8_SECRETS_KEY, roll the Deployment, and the server moves every row back to the old key at start. Then remove the previous key as in phase 3.

Never delete a sealed row​

A row the server cannot open is a row whose key is missing, not a broken row. The rows are the admin keys of your environments, the password of your directory bind account and the credentials of your alert channels. Deleting one loses that secret; restoring the key brings it back. When the start stops on a sealed row, find the key that sealed it and give it to the server as the previous key.

G8 dev