Tillered Docs | Access recovery

This guide covers the break-glass paths for regaining control of an Arctic agent when normal operator access is unavailable, and for clearing a cluster lock that is stuck. For recovering a peer whose host was lost or rebuilt, see backup and restore; the paths on this page assume the agent itself is intact.

Recovery token

Each agent generates a recovery token at startup and rotates it on a fixed interval. The token is an out-of-band credential that grants admin scope for a single request, intended for operators with local root access on the host.

Path: /etc/arctic/recovery.token
Contents: 32 random bytes, base64url-encoded
File mode: 0600, owned by the agent's user
Lifetime: 24 hours by default, tunable with RECOVERY_TOKEN_TTL

The token is presented to the agent in the X-Arctic-Recovery HTTP header. Any request carrying a matching token is granted admin scope and is logged at WARN so its use can be audited. The token does not establish a session: it authenticates each request that presents it, and it stays valid until the next rotation or agent restart, whichever comes first.

Changed in v1.4.2

The recovery token used to live until the next agent restart. It now expires: the agent rewrites /etc/arctic/recovery.token with a fresh value every RECOVERY_TOKEN_TTL (24 hours by default), and a presented token older than the TTL is rejected even if the rotation loop has died.

Read the file at time of use rather than keeping a copy; a saved token dies at the next rotation or restart. If recovery work runs long, re-read the file before each command. Restarting the agent once recovery work completes retires the token you used immediately instead of letting it age out.

When to use it

Reach for the recovery token when:

Operator credentials are lost. The client ID / secret for the cluster are gone and you need to authenticate to rotate them or create new credentials.
A credential is locked out. Five consecutive failed authentication attempts lock a credential out of the token endpoint for 15 minutes. The recovery token bypasses the token endpoint entirely, so it works even while a lockout is active.
You need to reach an api_access: internal peer. Internal-only peers reject user-facing endpoints from normal clients; the recovery token is the only way to drive operator operations against them.

Credentials created in a recovery-token session are not capped by a creating credential's scopes (there is none), so recovery can mint a full admin credential when the cluster's own admin credentials are gone.

Passing the token to the CLI

The CLI resolves the recovery token from three sources, in order of precedence:

--recovery-token <value> flag
$ARCTIC_RECOVERY_TOKEN environment variable
--recovery-token-file <path> flag

For example, reading the token straight off the host:

arctic --recovery-token-file /etc/arctic/recovery.token peers list

Or with the value inline:

ARCTIC_RECOVERY_TOKEN="$(sudo cat /etc/arctic/recovery.token)" \
  arctic credentials rotate

Recovery can be disabled

If the token path is not writable when the agent starts, recovery is disabled for that run; the agent does not fall back to open access. If the file becomes unwritable later, the current token simply ages out at the end of its TTL, so a read-only disk degrades to recovery-disabled, never to a long-lived credential. Internal-only mode then becomes strict and there is no break-glass path until the path is made writable and the agent is restarted.

The security boundary for the token is filesystem permissions alone. An attacker with local root on the host can read the token, but that attacker can already do anything to the host, so the token grants no additional reach.

Recovery-token lockout

The recovery-token header has the same brute-force protection as credentials: five consecutive requests with a bad X-Arctic-Recovery value lock the source out for 15 minutes, keyed by hashed source IP, and locked requests return HTTP 429. Locks are held in memory per agent, so they never span peers, expire after 15 minutes, and clear on an agent restart. With root on the host you are never locked out for long: read the current token from /etc/arctic/recovery.token and present it correctly, or restart the agent to clear all locks on that host.

Clearing a stuck cluster lock

Compose apply takes a cluster-wide lock so two operators do not apply conflicting changes at once. If an apply is interrupted (for example the machine running it is killed mid-run), the lock can be left held. Subsequent applies then fail with a contention error that includes the lock ID.

Release the stale lock with:

arctic state unlock <lock-id>

The contention error reports the lock ID to pass here. Useful flags:

Flag	Description
`--force`	Admin override on the cluster tier; bypasses the holder-id check.
`--cluster-only`	Release only the cluster lock, not the local file lock.
`--local-only`	Release only the local file lock, not the cluster lock.
`--state-dir <path>`	Override the `.arctic/` state directory location.

The unlock operation uses the cluster.lock scope. Only run it once you are sure no other apply is genuinely in progress; releasing a live lock can let two applies collide.

Recovering a lost peer host

Credential recovery and lock clearing assume the agent and its database are intact. When the host itself is gone (disk failure, accidental reprovision), the recovery unit is the peer's peer.key plus a database snapshot from /opt/tillered/backup/. With both, the peer is restored as itself in minutes; with only the key, it rejoins as a new identity and needs a short repair sequence. Both paths, and what to keep backed up off-host, are covered in backup and restore.

Access recovery