Backup and restore

An Arctic agent keeps two things on disk that matter for recovery: its identity and its state. Both are needed to restore a peer as itself. The identity alone is not enough: an agent that starts with a fresh database mints a new peer ID even when the old key is present, and the cluster treats it as a brand-new peer. This page describes what to back up, how the agent's automatic snapshots work, and the two recovery paths, in the order you want them: restore from a backup, and the fallback when only the key survived.

What to back up

Everything important lives in the data directory (/opt/tillered by default):

The pair to keep off-host is peer.key plus the newest snapshot. With both, a destroyed host is restored in minutes with no cluster surgery. With only the key, the peer comes back as a new identity and needs the manual repair described at the end of this page.

Automatic snapshots

The agent snapshots its database into $DATA_DIR/backup/ on a schedule, using SQLite's VACUUM INTO. Snapshots are taken while the agent runs, without stopping traffic, and each one is a compact, self-contained database file with no WAL or SHM sidecars.

Snapshots are named arctic.<UTC timestamp>.db, so a plain alphabetical sort is chronological. The agent also skips a snapshot when the newest one is younger than half the interval, so restart loops do not churn old recovery points out of the retention window.

To alert on backups going stale, watch the age of the newest file in $DATA_DIR/backup/: a newest snapshot older than the configured interval means the schedule is failing. Snapshot failures also surface as reconciler errors in the agent's journal; see Monitoring.

The snapshots only protect you if they leave the host. Copy the newest snapshot and peer.key to backup storage on a schedule, for example:

rsync -a --include='peer.key' --include='backup/***' --exclude='*' \
    /opt/tillered/ backup-host:/backups/arctic/$(hostname)/

Taking a snapshot on demand

arctic database snapshot captures a snapshot immediately on the agent the CLI is pointed at, bypassing the schedule's freshness check. Take one after a bootstrap, before an upgrade, or before risky maintenance:

arctic database snapshot
arctic database snapshot -o json    # machine-readable, prints the file path

The snapshot lands in the same $DATA_DIR/backup/ directory, counts toward the retention window, and works even when periodic snapshots are disabled. Remember it still needs to leave the host to be useful in a disaster.

Taking a manual backup

The cold path is a stopped copy. The agent checkpoints its write-ahead log on a clean stop, so the single database file is complete and consistent:

sudo systemctl stop arctic
sudo cp /opt/tillered/peer.key   /backup/arctic/peer.key
sudo cp /opt/tillered/arctic.db  /backup/arctic/arctic.db
sudo systemctl start arctic

If you need a hot copy and have the sqlite3 tool installed (it is not present on most minimal server images), SQLite's backup command also produces a consistent file:

sudo sqlite3 /opt/tillered/arctic.db ".backup '/backup/arctic/arctic.db'"

In most cases the automatic snapshots make both of these unnecessary.

Restoring a peer

On the rebuilt host, install the agent as usual, then stop it and put the two backed-up files in place. It does not matter that the fresh install already started once and generated a throwaway identity; the restore overwrites it. What matters is that you restore before running compose apply, so the throwaway identity never joins the cluster.

sudo systemctl stop arctic
sudo cp /backup/arctic/peer.key                 /opt/tillered/peer.key
sudo cp /backup/arctic/arctic.<timestamp>.db    /opt/tillered/arctic.db
sudo chown root:root /opt/tillered/peer.key /opt/tillered/arctic.db
sudo chmod 600       /opt/tillered/peer.key /opt/tillered/arctic.db
sudo systemctl start arctic

The agent starts as its original identity, with its membership, vouchers, and services intact. In live testing the other peers reported it healthy within one heartbeat cycle, and traffic through services targeting it resumed immediately. Changes made in the cluster after the snapshot was taken are re-learned through gossip over the following minutes.

If you run a hardened, non-root agent, set ownership of the restored files to the service user instead of root.

If you only have peer.key

Without a database, the agent generates a new peer ID on first start, even though it loads the old key. The cluster does not recognise the new ID: the rebuilt agent rejects heartbeats from its former cluster-mates, they reject its, and nothing recovers on its own. Every service that targets the old peer ID stays down until you repair it. What the key does preserve is the peer's fingerprint, so a fingerprint: pin in your compose file still verifies and does not need to be re-pinned.

Re-running compose apply in this state does not repair anything, and the CLI refuses it: it detects that the agent behind the peer's endpoint is a fresh install while the cluster still records an active member under that name, and aborts before joining anything. The repair is one command:

# The apply error names the dead member's ID, or find it
# with: arctic peers list
arctic peers replace <old-peer-id> ./cluster.yaml

peers replace verifies the agent behind the member's endpoint really is a fresh install, shows the services that will be deleted along with the dead member, and asks for one confirmation. It then removes the old member and re-applies the config, which joins the rebuilt agent under the same name and recreates its services. Add --dry-run to preview the repair without changing anything.

The same repair can be done manually, which is useful if you want to edit the config between the two halves:

# 1. Remove the dead old peer. This also deletes the services
#    that still reference it.
arctic peers delete <old-peer-id>

# 2. Join the rebuilt agent and recreate its services.
arctic compose apply ./cluster.yaml

Either way, expect the peer's services to be down from the moment the host was lost until the final apply completes.

If peer.key is also lost, the host cannot be restored as itself at all. Remove the old peer, update the fingerprint: pin for the host in your compose file to the new key's fingerprint, and re-apply.

Rolling back a compose change

compose apply captures a snapshot pair before it executes its plan, under .arctic/backup/ next to your config file: a copy of the pre-apply state cache plus a config export of the pre-apply cluster, filed under one timestamp. Capture happens before execution on purpose; a half-applied cluster is exactly when you need the snapshot. Only an apply that actually changes something captures one: a no-op apply does not, and neither does the first bootstrap of a fresh cluster, since there is no prior state to return to. Retention counts snapshots and is set by preferences.backup_retention (default 5) or --backup-retention per run. Backups written by pre-1.4.2 CLIs were state-only and list as non-rollbackable.

To undo a change, either re-apply the previous version of your cluster.yaml from version control, or roll back to a captured snapshot:

arctic state backups                  # list snapshots and their timestamps
arctic compose rollback --dry-run     # preview a rollback to the newest one
arctic compose rollback               # roll back to the newest snapshot
arctic compose rollback <timestamp>   # roll back to a named snapshot

A rollback runs through the standard apply pipeline with pruning enabled, so the cluster converges back to its pre-apply configuration: you get the usual plan preview and confirmation prompt, and the rollback captures its own pre-rollback snapshot first, so it can itself be undone. Peers are never pruned, so a rollback's blast radius is services and routes; a peer joined by the rolled-back apply stays a member, and you remove it with arctic peers delete if it should go.

Because compose is declarative, your version-controlled cluster.yaml is itself the source of truth: keeping it in Git gives you the full history to roll back to. See Compose.

See also

• Upgrades - back up arctic.db before upgrading
• Access recovery - break-glass access and stuck cluster locks
• Monitoring - watching agent health and logs
• Compose - declarative config as the source of truth