Backup and restore
Back up and restore an Arctic agent's identity and state, and roll back compose changes
An Arctic agent keeps two things on disk that matter for recovery: its identity and its state. Because cluster state is replicated between peers, an agent can often rebuild itself from the others, but backing up the identity is what lets you restore a peer as itself rather than re-adding it as a new one.
What to back up
Everything important lives in the data directory (/opt/tillered by default):
| Path | What it is | Why it matters |
|---|---|---|
peer.key | The peer's Ed25519 private key (its identity) | Without it, the host comes back as a new peer with a new ID, not the original |
arctic.db | The SQLite database: local config plus replicated cluster state | Restoring it brings the peer back with its known state immediately, before gossip re-syncs |
Back up peer.key on every peer and keep it somewhere safe; it is small and does not change. arctic.db changes continuously, so snapshot it on a schedule.
peer.key is the identity
Treat peer.key like a secret. Anyone who has it can impersonate the peer. If you lose it you cannot restore the peer as itself; you remove the old peer and add a new one in its place.
Taking a backup
Stop the agent so the database is consistent, copy the files, then start it again:
sudo systemctl stop arctic
sudo cp /opt/tillered/peer.key /backup/arctic/peer.key
sudo cp /opt/tillered/arctic.db /backup/arctic/arctic.db
sudo systemctl start arcticTo copy the database without stopping the agent, use SQLite's backup command so you capture a consistent file:
sqlite3 /opt/tillered/arctic.db ".backup '/backup/arctic/arctic.db'"
sudo cp /opt/tillered/peer.key /backup/arctic/peer.keyRestoring a peer
The replacement host needs the same IP address
The rest of the cluster reaches this peer at the address it had when it was backed up, so restore onto a host with the same IP address, reachable on the same endpoints. The identity in peer.key is necessary but not sufficient: the address has to match too, or the other peers cannot reach the restored one. If the replacement host will have a different address, restoring the identity does not help, because the cluster still cannot reach it. Add the new host as a new peer (a fresh bootstrap and identity) and remove the old one instead.
To bring a peer back on the same host, or a replacement that has the same IP address:
sudo systemctl stop arctic
sudo cp /backup/arctic/peer.key /opt/tillered/peer.key
sudo cp /backup/arctic/arctic.db /opt/tillered/arctic.db
sudo systemctl start arcticThe peer rejoins as its original identity and converges with the rest of the cluster over the next few gossip rounds. If you restore only peer.key and not the database, the peer keeps its identity and rebuilds its state from its peers.
If you run a hardened, non-root agent, set ownership of the restored files back to the service user before starting.
Rebuilding from the cluster
Because peers replicate state to each other, a peer that lost its database but kept peer.key and its address can rejoin and re-learn the cluster state through gossip. A full backup just makes the recovery immediate instead of waiting for convergence. See Recovery for break-glass access when credentials are lost.
Rolling back a compose change
compose apply snapshots cluster state before it makes changes, under .arctic/backup/ in its state directory. preferences.backup_retention (default 5) controls how many snapshots are kept. To undo a change, re-apply the previous version of your cluster.yaml:
arctic compose diff ./cluster.yaml # preview the rollback
arctic compose apply ./cluster.yaml # apply the previous desired stateBecause compose is declarative, your version-controlled cluster.yaml is itself the source of truth: keeping it in Git gives you the full history to roll back to. See Compose.