Troubleshooting

This page groups the problems operators hit most often into four areas: connectivity between agents, TLS and access errors, peer handshake failures, and configuration changes that do not take effect. Each entry lists the symptoms, the commands that narrow down the cause, and the fix. Where a problem has a distinctive error message, the entry is keyed on that literal text, so searching this page for the message should land on the entry.

Connectivity

Agent not responding

Symptoms: curl -sk https://AGENT_IP:8080/livez times out or is refused, and CLI commands fail with a connection error.

Check the service first:

systemctl status arctic

If it is not running, start it and read the recent log:

systemctl start arctic
journalctl -u arctic -n 50

If the service is up, confirm the agent is listening on the API port:

ss -tlnp | grep 8080

A healthy agent shows a listener owned by the arctic process:

LISTEN  0  4096  *:8080  *:*  users:(("arctic",...))

If the process is listening but the host still refuses the connection, a host firewall is the usual cause. These commands inspect the host's own firewall, which is separate from the tables Arctic manages:

# nftables
nft list ruleset | grep 8080

# iptables
iptables -L INPUT -n | grep 8080

# firewalld
firewall-cmd --list-ports

Open TCP 8080, or stop the conflicting firewall (see Prerequisites for the recommended host setup). If another process already holds port 8080, free it or set API_PORT to a different value before starting the agent.

Peers cannot communicate

Symptoms: handshakes fail, heartbeats do not arrive, or peers show as unhealthy.

Test the API path from one host to the other:

curl -sk https://PEER_IP:8080/livez

The IP tunnel carries non-TCP traffic over UDP 51840. Confirm that path is open as well:

nc -u PEER_IP 51840

If either fails, look at the route between the hosts:

traceroute PEER_IP
mtr PEER_IP

Both TCP 8080 and UDP 51840 must be reachable in both directions. When agents sit on different networks, check for NAT in the path and verify routing between the subnets.

Peers unreachable behind an egress proxy

Symptoms: hosts have a corporate egress proxy configured (HTTP_PROXY / HTTPS_PROXY); the handshake succeeds but the peer immediately shows unreachable, heartbeats never arrive, and everything works if the proxy variables are removed.

The agent honours the standard proxy environment variables for its outbound HTTP, so heartbeats, gossip, and voucher refresh are all sent to the proxy -- which typically cannot reach, or refuses, the private addresses of the other peers. The initial handshake dials direct, which is why a peer can join successfully and then go dark.

Confirm the unit is receiving proxy variables. A systemd service does not inherit shell environment, so these come from the systemd level (a DefaultEnvironment= in /etc/systemd/system.conf or a drop-in, often pushed by configuration management):

systemctl show arctic -p Environment
systemctl show-environment

Resolution: keep the proxy and exempt peer traffic from it with NO_PROXY, which accepts hostnames, IPs, and CIDRs. Add it as a drop-in so it survives agent upgrades:

sudo systemctl edit arctic

[Service]
Environment=NO_PROXY=10.0.0.0/8,192.168.0.0/16
Environment=no_proxy=10.0.0.0/8,192.168.0.0/16

sudo systemctl daemon-reload
sudo systemctl restart arctic

List every network your peers dial each other on. Set both cases of the variable, and note CIDR entries only match IP-literal endpoints; DNS-named peers must be listed by name or domain suffix. See the agent reference for the full behaviour, including how to clear the proxy for the agent entirely when it is not required for compliance.

The CLI is affected the same way: a workstation with proxy variables in its shell sends arctic API calls through the proxy too. Export the same NO_PROXY value in the shell (or CI environment) that runs the CLI.

Traffic is not being routed

Symptoms: a service exists but traffic does not flow, or packets are not being picked up by the proxy.

Confirm the service and its routes:

arctic services list
arctic services get SERVICE_ID

Inspect the firewall tables the agent commits to the kernel. TCP classification lives in the arctic table; tunnel marking lives in arctic_iptun:

nft list table inet arctic
nft list table inet arctic_iptun

You should see rules matching the source and destination CIDRs of your routes. If they are missing, check the firewall reconciler's log:

journalctl -u arctic | grep 'reconciler=firewall'

Resolution: verify the route CIDRs match the traffic you expect, force a cluster sync with arctic cluster sync, and confirm the source peer of the service is the agent you are testing from.

If the agent's rules and interfaces all look correct but routed traffic still vanishes on a cloud host, the drop may be happening at the hypervisor, before the packets ever reach the host: AWS discards traffic that is not addressed to the instance unless source/destination check is disabled, and Azure does the same unless IP forwarding is enabled on the NIC. See Cloud provider settings for the console and CLI steps. Nothing on the host will log these drops.

If the agent's rules are present but clients still fail with Packet filtered (an ICMP admin-prohibited reject) on a host that runs firewalld, the host firewall is rejecting the forwarded traffic before the agent's rules apply, because the agent's tpt-* and svc_* interfaces belong to no firewalld zone. See Keeping firewalld for the zone assignment that fixes it.

MACVLAN interface not created

Symptoms: a service sets requires_interface but no interface appears, or the interface exists without an address.

List interfaces and addresses:

ip link show
ip addr show

A service interface is named from the service ID, truncated to the kernel's 15-character limit, so look for a device matching the start of your service ID.

Check the network reconciler's log for the reason it was skipped or failed:

journalctl -u arctic | grep 'reconciler=network'

Resolution: confirm the host has a suitable parent interface, that the agent has CAP_NET_ADMIN (the systemd unit grants it), and that the interface name does not collide with an existing device.

DNS resolution

Symptoms: agents are unreachable by hostname, or lookups fail inside tunneled traffic.

nslookup HOSTNAME
dig HOSTNAME

Resolution: verify the host's resolvers, decide whether DNS should travel through Arctic at all, and add a route for the DNS server's IP if it should.

High latency

Symptoms: traffic through Arctic is slow, or round-trip times are high.

Compare a direct path against the tunneled path, and look for loss:

ping PEER_IP
mtr DESTINATION

Check whether a bandwidth limit is shaping the service:

arctic services get SERVICE_ID

Resolution: raise or remove the bandwidth_limit_mbps limit if it is too low, consider KCP transport on lossy links or for short, interactive flows, and rule out congestion on the underlying network.

Collecting debug information

When you open a support ticket, attach the output of:

arctic version
systemctl status arctic
journalctl -u arctic -n 100
ip addr show
ip route show
nft list table inet arctic
nft list table inet arctic_iptun
arctic peers list
arctic services list

TLS, trust, and access errors

v1.4.2 turned on peer TLS, identity pinning, and account lockout by default, which introduced a set of new refusals. Each entry below is keyed on the error it produces. The trust model behind them is covered in TLS and trust.

Client sent an HTTP request to an HTTPS server

Client sent an HTTP request to an HTTPS server.

This is expected as of v1.4.2 and does not mean the installation is broken. The agent now serves its API over TLS 1.3 by default, so a plain-HTTP request gets this response. Use the https form with -k:

curl -sk https://AGENT_IP:8080/livez

-k is needed because the certificate is self-signed; the agent's identity is verified by fingerprint pinning rather than a CA chain, so there is no certificate authority for curl to check against. The CLI handles TLS on its own and needs no flag. See TLS and trust.

One platform caveat: the curl that ships with macOS is built against LibreSSL, which cannot complete a handshake with the agent's Ed25519 certificate (it fails with exit code 35 before any HTTP exchange). Use the Arctic CLI for reachability checks from a Mac, or a Homebrew-installed curl (/opt/homebrew/opt/curl/bin/curl). Linux hosts are unaffected.

The same underlying error has a second face: a first-time compose apply against a fresh cluster that fails with read: connection reset by peer right after printing the plan, while the agent's journal logs client sent an HTTP request to an HTTPS server. That is the scheme-less endpoint bootstrap issue in v1.4.2 CLI builds from before the 2026-07-22 patch, which dialed the bootstrap over plaintext when an endpoint had no scheme. Update the CLI - patched builds read scheme-less endpoints as https://, and list the database command in arctic help - or write the endpoints in cluster.yaml with an explicit https:// scheme and re-run; see TLS and trust.

HTTP 429 slow_down from the token endpoint

{"error":"slow_down","error_description":"too many failed attempts; retry later"}

Account lockout. Five consecutive failed credential attempts lock that client_id out of the token endpoint for 15 minutes; further attempts return HTTP 429 without touching credential verification. The same protection applies per source IP to bad recovery-token headers.

The lock is held in memory on that agent only: it expires after 15 minutes, an agent restart clears it, and the other peers in the cluster are unaffected. If you are locked out of a credential, the recovery token still works because it bypasses the token endpoint entirely; see Access recovery.

Fix the credential rather than retrying: scripts that retry hard on bad credentials now convert an endless stream of 401s into a lockout.

Possible man-in-the-middle or key rotation

WARNING: the identity of 192.0.2.10 has changed.
  stored:    SHA256:jlPFJVDs837dHFX72m0SKh/R6IJuXHMy41lAzkyG3DQ
  presented: SHA256:5+k0XEMGuvXCr+Rcd7ucM66vn4VWfxuRe10IZv7VNG0
This is a possible man-in-the-middle, or a legitimate key rotation.
Re-run with --insecure to accept and store the new identity, or remove
the host from known_hosts.yaml.

A host the CLI already knows presented a different Ed25519 identity than the one pinned in its known-hosts store, and the connection was refused. The legitimate causes are a host rebuild or a deliberate key rotation; the malicious one is an attacker interposed on the path.

Verify out-of-band before accepting: read the fingerprint on the trusted host with arctic-agent fingerprint, or from the identity_fingerprint field the agent logs at boot (journalctl -u arctic). If it matches what the CLI reported, the change is genuine; re-run the command with --insecure to accept and re-learn the identity. If it does not match, stop and investigate the network path.

CLI exits 3 naming known_hosts.yaml

Error: could not load the known-hosts trust store: trust: parse known hosts
~/.config/arctic/known_hosts.yaml: yaml: line 1: did not find expected ',' or ']'
Identity pinning cannot run without it. Inspect or remove the file, or re-run
with --insecure to proceed unpinned

The CLI's trust store (~/.config/arctic/known_hosts.yaml) could not be read or parsed, and the CLI fails closed rather than silently running with identity pinning disabled. Restore the file from a backup, or run once with --insecure, the deliberate unpinned override, to re-learn identities and rewrite the store.

Older CLI exits 3 naming two state versions

Error: state file .arctic/cluster.state has on-disk schema_version 2 which is newer than supported version 1. Upgrade arctic.

A downgraded CLI read a .arctic/cluster.state file written by v1.4.2 or later, which moved the state file to schema version 2. Upgrade the CLI, or remove the .arctic/ directory and re-bootstrap the local state with the next compose apply.

out-of-band drift detected: peer ...

out-of-band drift detected: peer ...

Before v1.4.2 this was a false positive on effectively every compose apply and diff: the drift hash included the peer endpoint list, which changes out-of-band by design. That is fixed; the hash now covers only the peer name and Ed25519 public key. A peer drift report from a v1.4.2 CLI therefore signals a real identity change - a rebuilt host or a replaced key - and should be checked against the host's fingerprint, not dismissed.

Handshake failures

How a handshake works

When you add a peer, the two agents run a challenge-response handshake before they trust each other:

1. The initiator sends a 32-byte random challenge.
2. Both sides sign a message built from the challenge, the peer IDs, and the license ID.
3. Each side verifies the other's signature against the public key it holds.

Only after this succeeds do the peers accept each other's gossip. A failed handshake is retried up to three times, after which the peer is marked unreachable until something changes. See Clustering for the full trust model.

Common errors

Connection refused

Error: handshake failed: connection refused

The TCP connection to the remote agent could not be made. Confirm the remote agent is up (curl -sk https://REMOTE_IP:8080/livez), that the host is reachable (ping REMOTE_IP), and that TCP 8080 is open.

Connection timeout

Error: handshake failed: connection timeout

A path exists but the connection does not complete. Look for a firewall dropping packets, a NAT in the way, or the remote agent listening on a different interface than the one you are reaching.

License mismatch

Error: handshake failed: license mismatch

The two agents were bootstrapped with licenses that carry different customer identities, so they refuse to join the same cluster. Compare the license on each agent:

# Local agent
arctic license status

# Remote agent
arctic license status --url https://REMOTE_IP:8080

If they differ, re-bootstrap one agent with the correct license.

Invalid signature

Error: handshake failed: invalid signature

The peer's signature did not verify against the keys it should have. This points to a corrupted or replaced peer key. Re-bootstrap the affected agent, and contact support if it recurs.

Peer already exists

Error: peer already exists in cluster

The peer is already in the cluster. List peers to confirm:

arctic peers list

If you genuinely need to re-add it, delete it first:

arctic peers delete PEER_ID --yes

Node limit exceeded

Error: handshake failed: node limit exceeded

The license caps the number of nodes and the cluster is at that cap. Check the limit:

arctic license status

Remove unused peers to free a slot, or update to a license with a higher node count.

Debugging steps

Run the failing command with debug output, or trace the HTTP exchange:

arctic peers add REMOTE_IP:8080 --debug
arctic peers add REMOTE_IP:8080 --trace

Watch the logs on both agents while the handshake runs:

# Local agent
journalctl -u arctic -f

# Remote agent
ssh user@REMOTE_IP journalctl -u arctic -f

Read the remote agent's cluster identity. This endpoint needs no authentication, which makes it a quick way to confirm what cluster the remote agent thinks it belongs to:

curl -sk https://REMOTE_IP:8080/v1/cluster/identity

{
  "cluster_id": "clu_01HABCDEF456...",
  "customer_id": "cust_01HXYZABC123...",
  "license_id": "lic_01HXYZDEF789...",
  "peer_id": "peer_01HXYZGHI012...",
  "public_key": "XdguO3mIzx3BFaxwxhdhXxvQQwndvug0M9Z5reLpYRI=",
  "version": "arctic/v1.4.2"
}

An agent that has not yet been bootstrapped returns {"error":{"code":"PEER_NOT_BOOTSTRAPPED","message":"not bootstrapped"}} instead.

Confirm license_id matches the rest of your cluster. A handshake needs traffic in both directions, so test the reverse path too:

# Local to remote
curl -sk https://REMOTE_IP:8080/livez

# Remote to local
ssh user@REMOTE_IP curl -sk https://LOCAL_IP:8080/livez

Firewall requirements

If agents sit behind NAT, forward TCP 8080 to each agent, give the public address when you add the peer, and keep the mapping stable.

Recovery steps

If handshakes keep failing after the checks above, restart both agents:

systemctl restart arctic

As a last resort, re-bootstrap an agent. This drops its local database and all state stored only on that node:

systemctl stop arctic
rm /opt/tillered/arctic.db
systemctl start arctic
arctic bootstrap --url https://localhost:8080 --license-file license.json

If the problem survives a re-bootstrap, contact support.

Configuration not applied

How configuration is applied

A change you make through the API or arctic compose apply does not touch the kernel directly. It flows like this:

1. The change is written to the agent's SQLite database.
2. The write fires an event, and the relevant reconciler wakes up.
3. The reconciler computes the desired state and applies it: the network reconciler manages MACVLAN interfaces, the firewall reconciler commits nftables rules over netlink, and the TProxy and IP-tunnel reconcilers push fresh config into their in-process engines.

There are no generated config files and nothing to reload by hand. If applied state drifts from the database, the fix is to get the reconciler to run again, not to edit a file.

Symptoms

• A service was created but traffic is not routed.
• Routes were updated but the old routing still applies.
• A bandwidth limit is not taking effect.
• A requires_interface service has no interface.

Diagnosis

Force a cluster sync and give it a few seconds:

arctic cluster sync

curl -sk -X POST https://AGENT_IP:8080/v1/cluster/sync \
  -H "Authorization: Bearer $TOKEN"

Note that as of v1.4.2 a no-op compose apply no longer re-seeds peer endpoint lists as a side effect; if you were relying on an apply to push endpoints back out, arctic cluster sync is the command that forces that round.

Read the reconciler logs. Each reconciler tags its log lines with a reconciler field, so you can filter to the one you care about:

journalctl -u arctic | grep -E 'reconciler=(network|firewall|tproxy|iptun)'

Inspect the kernel state the agent should have produced:

# Firewall classification and tunnel marking
nft list table inet arctic
nft list table inet arctic_iptun

# Service interfaces
ip link show

Common issues

Firewall rules missing

Symptoms: nft list table inet arctic does not show the rules you expect.

The agent owns these tables and rewrites them on every reconcile; you do not load them yourself. Look for an error in the firewall reconciler's log, then restart the agent to force a full rebuild:

journalctl -u arctic | grep 'reconciler=firewall'
systemctl restart arctic

TProxy engine not applying config

Symptoms: TCP routing reflects old service definitions.

Check the TProxy reconciler and engine log, then restart to re-apply from a clean state:

journalctl -u arctic | grep 'reconciler=tproxy'
systemctl restart arctic

IP tunnel not applying config

Symptoms: tunnels for non-TCP traffic are not established.

The tunnel runs inside the agent, so there is no separate interface or service to inspect. Confirm UDP 51840 is reachable between the peers, then read the reconciler log:

nc -u PEER_IP 51840
journalctl -u arctic | grep 'reconciler=iptun'

Restart the agent if the log shows the engine failing to start or apply.

MACVLAN interface missing

Symptoms: a requires_interface service has no interface.

journalctl -u arctic | grep 'reconciler=network'

Confirm the parent interface exists and that the interface name does not collide with an existing device.

Applied state does not match the database

Sometimes the database holds the right data but the kernel does not reflect it. Confirm what the database actually contains:

arctic services list --json
arctic routes list --service SERVICE_ID --json

If the data is correct, the reconciler either errored or never ran. Restart the agent to force every reconciler through a full pass:

systemctl restart arctic

How long changes take

A change normally applies within a second or two: the database write fires an event and the reconciler runs immediately. As a backstop, every core reconciler also resyncs on a 60-second timer, so a dropped event still self-corrects within a minute. Cluster-wide changes additionally need a gossip round to reach other peers; arctic cluster sync forces that round instead of waiting for the next heartbeat.

Collecting debug information

journalctl -u arctic --since "10 minutes ago"
arctic services list --json
arctic routes list --service SERVICE_ID --json
nft list table inet arctic
nft list table inet arctic_iptun
systemctl status arctic

See also

• TLS and trust - peer TLS, fingerprints, and the known-hosts store
• Upgrades - upgrading agents and the CLI
• Access recovery - break-glass access and clearing a stuck cluster lock
• Clustering - the trust and gossip model behind handshakes