Peer management

Add a peer

This section shows you how to add a new Arctic agent to an existing cluster.

Before you start

Ensure you have:

• An existing Arctic cluster with at least one bootstrapped agent
• A new agent that is running and accessible
• The new agent bootstrapped with the same license as your cluster
• Network connectivity between the existing cluster and the new agent

Steps

1. Bootstrap the new agent

If the new agent has not been bootstrapped yet, initialize it with your license:

arctic bootstrap \
  --url https://NEW_AGENT_IP:8080 \
  --license-file license.json \
  --name "new-agent"

curl -k -X POST https://NEW_AGENT_IP:8080/v1/bootstrap \
  -H "Content-Type: application/json" \
  -d '{"license": LICENSE_JSON, "name": "new-agent"}'

2. Verify the new agent is healthy

Confirm the agent is running and ready:

curl -sk https://NEW_AGENT_IP:8080/livez

Expected response: {"status":"ok",...}

3. Add the peer

From your CLI (configured to connect to an existing cluster agent), add the new peer by its address:

arctic peers add NEW_AGENT_IP:8080

arctic peers add NEW_AGENT_IP:8080 --name "new-agent"

# Get an access token first
TOKEN=$(curl -sk -X POST https://EXISTING_AGENT_IP:8080/v1/oauth/token \
  -d "grant_type=client_credentials&client_id=cli_YOUR_ID&client_secret=sec_YOUR_SECRET" \
  | jq -r '.access_token')

# Add the peer. Prefer the endpoints list over the deprecated
# single "address" field; the agent tries each entry in order.
curl -k -X POST https://EXISTING_AGENT_IP:8080/v1/peers \
  -H "Authorization: Bearer $TOKEN" \
  -H "Content-Type: application/json" \
  -d '{"endpoints": ["NEW_AGENT_IP:8080"], "name": "new-agent"}'

4. Verify the peer was added

List all peers to confirm the new agent appears:

arctic peers list

arctic peers list --remote

curl -k -X GET https://EXISTING_AGENT_IP:8080/v1/peers \
  -H "Authorization: Bearer $TOKEN"

The new peer should appear with is_local: false.

What happens during peer addition

When you add a peer, Arctic performs these steps:

1. Identity exchange: The existing agent contacts the new agent over TLS and they exchange public keys. First contact uses identity binding: the new agent's TLS certificate key must match the identity its signed handshake response authenticates, so the transport cannot be intercepted by a third party presenting its own certificate. See TLS and trust.
2. Signature verification: Both agents run an Ed25519 challenge-response and validate license compatibility.
3. Database update: Both agents store each other's peer information.
4. Gossip initialization: The agents begin exchanging heartbeats and synchronize state.

The handshake response includes a transitive-trust voucher. The process typically completes within seconds.

Troubleshooting

Handshake failed: connection refused

The new agent is not accessible. Check:

• Network connectivity: curl -sk https://NEW_AGENT_IP:8080/livez
• Firewall rules allow port 8080
• The agent service is running: systemctl status arctic

Handshake failed: license mismatch

The agents have different licenses. Verify:

• Both agents were bootstrapped with the same license file
• Check license status on both agents: arctic license status

Peer already exists

The peer is already in the cluster. This can happen if:

• You previously added this peer
• Another cluster member already discovered this peer via gossip

Use arctic peers list to see existing peers.

Node limit exceeded

Your license has a maximum node limit. Check your license:

arctic license status

Contact your administrator to upgrade your license if needed.

Update a peer

This section shows you how to update a peer's metadata, such as its human-readable name or QoS toggle.

Before you start

Ensure you have:

• The peer ID of the peer you want to update
• Appropriate permissions (peers.write scope)

Steps

1. Find the peer ID

List peers to find the ID:

arctic peers list

curl -k -X GET https://AGENT_IP:8080/v1/peers \
  -H "Authorization: Bearer $TOKEN"

Peer IDs are prefixed with peer_.

2. Update the peer

Update the peer's name:

arctic peers update PEER_ID --name "prod-svr-1"

arctic peers update PEER_ID --qos=false

curl -k -X PUT https://AGENT_IP:8080/v1/peers/PEER_ID \
  -H "Authorization: Bearer $TOKEN" \
  -H "Content-Type: application/json" \
  -d '{"name": "prod-svr-1", "qos_enabled": false}'

Both name and qos_enabled are optional; omit a field to leave it unchanged.

3. Verify the update

Confirm the change:

arctic peers get PEER_ID

curl -k -X GET https://AGENT_IP:8080/v1/peers/PEER_ID \
  -H "Authorization: Bearer $TOKEN"

What can be updated

Other peer attributes (public key, peer ID, etc.) are immutable and cannot be changed.

Propagation

A name change made through the CLI is also synced to your local CLI config. Other cluster members maintain their own view of peer metadata. If you want consistent naming across the cluster, update the peer on each agent.

Troubleshooting

Peer not found

If the peer ID is not found:

1. Verify the peer ID: arctic peers list
2. Check you are connected to the correct agent

Permission denied

If you receive an authorization error:

1. Verify your credentials have the peers.write scope
2. Check that your token has not expired

Remove a peer

This section shows you how to gracefully remove an agent from your Arctic cluster.

Before you start

Understand the difference between removing a remote peer and removing the local peer:

• Remote peer: Removes another agent from your view of the cluster.
• Local peer (remove-self): Gracefully removes the current agent from the entire cluster.

The local peer cannot be removed with peers delete. Use peers remove-self for the local agent.

Remove a remote peer

Use this when you want to remove another agent from your cluster.

1. Find the peer ID

List all peers to find the ID of the peer you want to remove:

arctic peers list

curl -k -X GET https://AGENT_IP:8080/v1/peers \
  -H "Authorization: Bearer $TOKEN"

2. Delete the peer

Remove the peer by ID:

arctic peers delete PEER_ID

# You will be prompted to confirm. Use --yes to skip confirmation:
arctic peers delete PEER_ID --yes

curl -k -X DELETE https://AGENT_IP:8080/v1/peers/PEER_ID \
  -H "Authorization: Bearer $TOKEN"

3. Verify removal

Confirm the peer no longer appears in the list:

arctic peers list

Remove the local peer (self-removal)

Use this when you want to gracefully remove the current agent from the cluster. This is useful when decommissioning a node.

1. Remove self

arctic peers remove-self

# You will be prompted to confirm. Use --yes to skip confirmation:
arctic peers remove-self --yes

arctic remove-self --yes

curl -k -X DELETE https://AGENT_IP:8080/v1/peers/self \
  -H "Authorization: Bearer $TOKEN"

2. What happens

When you remove self:

1. The local peer is marked as deactivated.
2. A signed deactivation announcement is broadcast to all cluster peers.
3. Other peers receive the announcement and remove this peer from their databases.
4. The agent shuts down gracefully.

Cleanup after removal

Deleting a peer also tombstones every service whose source or target is that peer, and the deletions propagate to the rest of the cluster via gossip. There is no separate service cleanup step. If the peer was removed because its host was rebuilt and you want the services back, see arctic peers replace in the CLI reference and Backup and restore: it deletes the dead member and re-applies your compose config in one step.

After removing a peer, you may want to:

Update CLI configuration

If you have the removed peer in your CLI config, remove it:

arctic config delete-peer PEER_ID_OR_NAME

Troubleshooting

Cannot delete local peer

If you try to delete the local peer using peers delete:

Error: cannot delete local peer (use 'arctic peers remove-self' instead)

Use arctic peers remove-self for the local agent.

Peer not found

If the peer ID is not found:

1. Verify the peer ID: arctic peers list
2. The peer may have already been removed by another cluster member

Services still listed after removal

Deleting a peer tombstones its services on the peer you ran the delete against; the other members pick the tombstones up via gossip. If a service still appears on another member, give gossip a minute to converge before investigating.