Tillered Docs | Monitoring

Arctic exposes its health through unauthenticated health endpoints, status commands, and structured logs. Use these to watch a single agent, see the health of the whole cluster, and feed an external uptime or log-aggregation system.

Health endpoints

Every agent serves two health endpoints on the API port (8080 by default) with no authentication, so a load balancer or uptime check can poll them directly. As of v1.4.2 the API port speaks TLS 1.3 with a self-signed certificate, so poll over https and skip certificate verification (-k for curl); agent identity is verified by fingerprint pinning rather than a CA chain. See TLS and trust.

Endpoint	Answers	Use for
`GET /livez`	Is the agent process up?	Liveness probes, uptime checks
`GET /readyz`	Is the agent ready to serve and route?	Readiness probes before sending traffic

curl -sk https://AGENT_IP:8080/livez

{"status":"ok","timestamp":"2026-01-15T10:30:00.108306567Z"}

/readyz returns the same shape plus per-check detail and is non-200 while the agent is still starting or a dependency is not ready. Point a liveness probe at /livez and a readiness probe at /readyz.

Checking one agent

arctic health reports on the agent your CLI is pointed at:

arctic health            # liveness
arctic health --readyz   # readiness, with per-check detail

It exits non-zero (code 4) when the agent is unreachable, so it works in scripts and CI.

Checking the cluster

arctic cluster status gives a cluster-wide view from the agent you query: each peer's last-seen time, reachability, and registry hashes, plus the local registry totals. Use it to spot a peer that is unreachable or has not converged.

arctic cluster status

Registry hashes that differ between peers mean state has not converged yet. It usually resolves on the next gossip round, or you can force one with arctic cluster sync. See Clustering for how convergence works.

Logs

The agent logs to the journal. Structured JSON (the default, LOG_FORMAT=json) is the right choice when you ship logs to an aggregator; set LOG_FORMAT=text for readable local output.

# Follow the log
journalctl -u arctic -f

# Recent entries
journalctl -u arctic -n 100 --since "1 hour ago"

Each reconciler tags its lines with a reconciler field (firewall, network, tproxy, iptun), so you can filter to one subsystem:

journalctl -u arctic | grep 'reconciler=firewall'

Raise LOG_LEVEL to debug temporarily for deeper detail; leave it at info in production.

Audit events

As of v1.4.2 the agent writes a security audit trail into the same log stream. Two events carry it:

Every REST 403 emits a WARN entry with event=authz_denied: the authenticated subject, the operation it attempted, the required versus held scopes, and a hashed source IP. Denied requests previously left no trace, so scope probing was invisible.
Every committed configuration change (service, route, or peer) emits an INFO entry with event=config_change: the action, the entity, the acting identity (actor plus actor_type: operator credential, peer, recovery token, or system), the CRDT version transition, and a bounded before/after of the changed fields. Key material never appears in the trail.

The trail is complete across all three write paths. A change made directly on a node records the operator's client ID; a change forwarded to the authoritative source peer is recorded on both nodes, with forwarded_to on the node the operator hit; and a change arriving via gossip is recorded on every receiving node with the verified signing peer as the actor and origin=gossip. Steady-state heartbeats emit nothing, so the audit stream carries changes, not noise.

Audit events are stable structured JSON keyed by the event field, which makes them suitable for SIEM ingestion: alert on authz_denied bursts and on config_change entries whose actor you do not recognise.

A few other lines are worth watching for: event=account_locked when a credential or recovery-token source crosses the lockout threshold, event=recovery_token_rotated on each recovery-token rotation, and the boot-time identity_fingerprint line that records the identity the agent came up with.

What to watch

Liveness: /livez on every agent. A failure means the process is down.
Readiness: /readyz before routing traffic to an agent.
Peer reachability: arctic cluster status for peers stuck unreachable or with stale last-seen times.
License state: an expiring license changes what the agent allows. Check it with arctic license status; see Licensing.
Log errors: WARN and ERROR lines, especially repeated reconciler errors.
Audit events: authz_denied bursts (scope probing or a misconfigured client) and config_change entries from actors you do not recognise.

Planned: Prometheus metrics

Metric-based monitoring through a Prometheus endpoint is planned. Until then, use the health endpoints, status commands, and logs described above.

Monitoring