Peer management

Add a peer

This section shows you how to add a new Arctic agent to an existing cluster.

Before you start

Ensure you have:

• An existing Arctic cluster with at least one bootstrapped agent
• A new agent that is running and accessible
• The new agent bootstrapped with the same license as your cluster
• Network connectivity between the existing cluster and the new agent

Steps

1. Bootstrap the new agent

If the new agent has not been bootstrapped yet, initialize it with your license:

arctic bootstrap \
  --url http://NEW_AGENT_IP:8080 \
  --license-file license.json \
  --name "new-agent"

curl -X POST http://NEW_AGENT_IP:8080/v1/bootstrap \
  -H "Content-Type: application/json" \
  -d @license.json

2. Verify the new agent is healthy

Confirm the agent is running and ready:

curl http://NEW_AGENT_IP:8080/livez

Expected response: {"status":"ok",...}

3. Add the peer

From your CLI (configured to connect to an existing cluster agent), add the new peer:

arctic peers add NEW_AGENT_IP:8080

# Get access token first
TOKEN=$(curl -s -X POST http://EXISTING_AGENT_IP:8080/v1/oauth/token \
  -d "grant_type=client_credentials&client_id=YOUR_CLIENT_ID&client_secret=YOUR_CLIENT_SECRET" \
  | jq -r '.access_token')

# Add the peer
curl -X POST http://EXISTING_AGENT_IP:8080/v1/peers \
  -H "Authorization: Bearer $TOKEN" \
  -H "Content-Type: application/json" \
  -d '{"address": "NEW_AGENT_IP:8080"}'

4. Verify the peer was added

List all peers to confirm the new agent appears:

arctic peers list

curl -X GET http://EXISTING_AGENT_IP:8080/v1/peers \
  -H "Authorization: Bearer $TOKEN"

The new peer should appear in the list with LOCAL: false.

What happens during peer addition

When you add a peer, Arctic performs these steps:

1. Identity Exchange: The existing agent contacts the new agent and they exchange public keys
2. Signature Verification: Both agents verify the signatures match their shared license
3. Database Update: Both agents store each other's peer information
4. Gossip Initialization: The agents begin exchanging heartbeats and will synchronize state

The entire process typically completes within seconds.

Troubleshooting

Handshake Failed: Connection Refused

The new agent is not accessible. Check:

• Network connectivity: curl http://NEW_AGENT_IP:8080/livez
• Firewall rules allow port 8080
• The agent service is running: systemctl status arctic

Handshake Failed: License Mismatch

The agents have different licenses. Verify:

• Both agents were bootstrapped with the same license file
• Check license IDs: arctic license show on both agents

Peer Already Exists

The peer is already in the cluster. This can happen if:

• You previously added this peer
• Another cluster member already discovered this peer via gossip

Use arctic peers list to see existing peers.

Node Limit Exceeded

Your license has a maximum node limit. Check your license:

arctic license show

Contact your administrator to upgrade your license if needed.

Update a peer

This section shows you how to update a peer's metadata, such as its human-readable name.

Before you start

Ensure you have:

• The peer ID of the peer you want to update
• Appropriate permissions (peers.write scope)

Steps

1. Find the peer ID

List peers to find the ID:

arctic peers list

curl -X GET http://AGENT_IP:8080/v1/peers \
  -H "Authorization: Bearer $TOKEN"

2. Update the peer

Update the peer's name:

arctic peers update PEER_ID --name "prod-svr-1"

curl -X PUT http://AGENT_IP:8080/v1/peers/PEER_ID \
  -H "Authorization: Bearer $TOKEN" \
  -H "Content-Type: application/json" \
  -d '{"name": "Production Server 1"}'

3. Verify the update

Confirm the change:

arctic peers get PEER_ID

curl -X GET http://AGENT_IP:8080/v1/peers/PEER_ID \
  -H "Authorization: Bearer $TOKEN"

What can be updated

Currently, the following peer metadata can be updated:

Other peer attributes (public key, peer ID, etc.) are immutable and cannot be changed.

Propagation

Peer name updates are local to the agent where you make the change. Other cluster members maintain their own names for peers. If you want consistent naming across the cluster, update the peer on each agent.

Troubleshooting

Peer Not Found

If the peer ID is not found:

1. Verify the peer ID: arctic peers list
2. Check you are connected to the correct agent

Permission Denied

If you receive an authorization error:

1. Verify your credentials have the urn:tillered:arctic:peers.write scope
2. Check that your token has not expired

Remove a peer

This section shows you how to gracefully remove an agent from your Arctic cluster.

Before you start

Understand the difference between removing a remote peer and removing the local peer:

• Remote peer: Removes another agent from your view of the cluster
• Local peer (remove-self): Gracefully removes the current agent from the entire cluster

Remove a remote peer

Use this when you want to remove another agent from your cluster.

1. Find the peer ID

List all peers to find the ID of the peer you want to remove:

arctic peers list

curl -X GET http://AGENT_IP:8080/v1/peers \
  -H "Authorization: Bearer $TOKEN"

2. Delete the peer

Remove the peer by ID:

arctic peers delete PEER_ID

# You will be prompted to confirm. Use --yes to skip confirmation:
arctic peers delete PEER_ID --yes

curl -X DELETE http://AGENT_IP:8080/v1/peers/PEER_ID \
  -H "Authorization: Bearer $TOKEN"

3. Verify removal

Confirm the peer no longer appears in the list:

arctic peers list

Remove the local peer (self-removal)

Use this when you want to gracefully remove the current agent from the cluster. This is useful when decommissioning a node.

1. Remove self

arctic peers remove-self

# You will be prompted to confirm. Use --yes to skip confirmation:
arctic peers remove-self --yes

curl -X DELETE http://AGENT_IP:8080/v1/peers/self \
  -H "Authorization: Bearer $TOKEN"

2. What happens

When you remove self:

1. The local peer is marked as deactivated
2. A signed deactivation announcement is broadcast to all cluster peers
3. Other peers receive the announcement and remove this peer from their databases
4. Services involving this peer become inactive

Cleanup after removal

After removing a peer, you may want to:

Delete associated services

Services involving the removed peer will no longer function. List and delete them:

# List services involving the removed peer
arctic services list --target-peer REMOVED_PEER_ID

# Delete each service
arctic services delete SERVICE_ID --yes

Update CLI configuration

If you have the removed peer in your CLI config, remove it:

arctic config delete-peer PEER_ID_OR_NAME

Troubleshooting

Cannot Delete Local Peer

If you try to delete the local peer using peers delete:

Error: cannot delete local peer, use 'peers remove-self' instead

Use arctic peers remove-self for the local agent.

Peer Not Found

If the peer ID is not found:

1. Verify the peer ID: arctic peers list
2. The peer may have already been removed by another cluster member
3. Use --all-contexts to check all configured agents

Services Still Active After Removal

Services are not automatically deleted when a peer is removed. You must manually delete services that reference the removed peer.