Release Notes

Release notes for the self-hosted Arctic agent and CLI. Newest releases are listed first.

1.4.2

A release for multi-NIC hosts. Until now Arctic auto-detected a single egress interface per host, so on a machine with more than one NIC, or an overlay device such as Tailscale or WireGuard, you could not say which device a route matches on, which NIC a service's MACVLAN is built on, or which interface a service's traffic leaves by. Three new compose fields make each of those an explicit per-service choice, and a per-service Multipath TCP opt-in lets one TCP tunnel spread across more than one of those paths at once. See the new Uplinks guide.

This release also hardens cluster security across the board. Agent-to-agent traffic and the operator API now run over TLS 1.3 with each peer's identity pinned, on by default; every authorization denial and committed configuration change is now a structured audit event; repeated bad credentials lock out; and credentials are explicitly scoped rather than implicitly admin.

Finally, 1.4.2 fixes two firewall defects that caused stale routing state to accumulate on cluster changes, along with several TProxy and IP-tunnel data-plane bugs.

Highlights

• Pick the network path per service: routes[].device matches a route on a specific inbound interface, interface.parent_dev pins the MACVLAN parent, and uplink_dev pins the egress device a service's traffic leaves by. All three accept overlay and VPN interfaces, so a service can be forced over a Tailscale or WireGuard lane with one line. See Uplinks.
• Per-service Multipath TCP: mptcp: {enabled: true} opts a TCP-transport service's tunnel dials into MPTCP, with an optional subflows count, so one tunnel can use more than one NIC at once. Dials fall back to plain TCP when the kernel or the remote peer lacks MPTCP support.
• Peer mesh over TLS by default: The agent serves and dials the mesh, and the operator REST API, over TLS 1.3 with each peer's Ed25519 identity pinned. The CLI learns and pins identities on first contact, and a fingerprint: per peer in the compose file verifies identity before first contact. See TLS and trust.
• Security hardening: A full audit trail for denials and configuration changes, account lockout after repeated failures, a recovery token that rotates instead of living until restart, and scoped, capped credentials.
• Automatic database snapshots: The agent snapshots arctic.db into $DATA_DIR/backup/ on a schedule, safely while running. A snapshot plus peer.key restores a peer as itself after a host rebuild. See Backup and restore.
• Safer compose runs: Apply asks for consent before any mutation, captures a rollback point before executing (arctic compose rollback restores it), and refuses to join a rebuilt peer over an existing member, with arctic peers replace as the one-step repair.
• Firewall no longer accumulates routing state: Stale rules and routes left behind on cluster changes are now cleaned up automatically.

Upgrading to 1.4.2

This is not a drop-in upgrade like 1.4.1; the first note changes the upgrade procedure itself, and several credential behaviors are breaking. The new agent environment variables mentioned below are catalogued in the Agent configuration reference.

• Upgrade every node in a cluster together (breaking): Peer TLS is on by default and cluster-wide; a dialing peer expects a TLS peer, so there is no rolling upgrade path. The CLI discovers the change on its own: the first apply or diff after the upgrade probes each host, learns its TLS identity, and rewrites saved http:// endpoints. Set PEER_TLS_ENABLED=false on every agent to keep plaintext, for example behind a TLS-terminating proxy.
• Database migrations run automatically: The agent migrates its database the first time 1.4.2 starts, adding the device-selection and MPTCP fields. The matching compose fields are optional; existing files apply unchanged. Take a backup beforehand as usual.
• Scope-less credential creation now fails (breaking): Creating a credential with no scopes returns 400 instead of silently minting an admin credential, and arctic credentials create requires --scopes (the read, write, and admin aliases are accepted). Scripts that relied on the implicit admin default must pass --scopes.
• Credential grants are capped by the creator's scopes (breaking): A credential can no longer mint scopes it does not hold. Mint privileged credentials from an admin credential.
• The CLI config holds no admin credential after bootstrap (breaking for credential workflows): Bootstrap saves a scoped working credential to the config and writes the admin pair to break-glass.<cluster-id>.json next to it. Credential management and other admin-only operations need that pair via --client-id/--client-secret or ARCTIC_CLIENT_ID/ARCTIC_CLIENT_SECRET. Pass --no-scoped-credential at bootstrap to keep the old behavior.
• Revoked credentials cut off live tokens within about 30 seconds: Tooling that expects outstanding tokens to work until expiry now sees 401s within the REVOCATION_CACHE_TTL window. Mint a replacement credential before revoking the old one.
• Repeated auth failures lock out: Five consecutive bad credential attempts return HTTP 429 for 15 minutes; the same applies per source IP to bad recovery-token headers. Locks are per agent, in memory, and clear on restart; the recovery token bypasses the token endpoint entirely. Scripts that retry hard on bad credentials will hit this. See Troubleshooting.
• The recovery token rotates every 24 hours: Never store a copy of /etc/arctic/recovery.token; read it at time of use. A token copied before the last rotation or restart no longer authenticates. RECOVERY_TOKEN_TTL changes the interval. See Access recovery.
• The state directory anchors to the config file: .arctic/ now lives next to the compose file rather than in the working directory. Applies run from the config's own directory see no change; others get a one-time move notice (the fallback is removed in v1.4.3).
• The trust store no longer follows --config: known_hosts.yaml always lives in the user config home now. Setups that kept it inside a custom config directory should pass --known-hosts or set ARCTIC_KNOWN_HOSTS.
• Audit log volume: Expect one INFO line per committed configuration change per node and one WARN line per denied request. If you ship logs to a SIEM, the new events are stable structured JSON keyed by event.

Security and trust

• Peer TLS with Ed25519 pinning: Each agent presents an in-memory self-signed certificate derived from its peer.key (no CA, no issuance) and every agent-to-agent dial is pinned to the peer's identity key. First contact with a TLS cluster needs no ceremony: scheme-less endpoints are probed, verified, and upgraded to https automatically.
• Trust-on-first-use in the CLI, with a provable upgrade path: The CLI records each agent's identity in ~/.config/arctic/known_hosts.yaml on first contact and pins it thereafter; a changed identity is refused with an SSH-style warning. To remove even the first-contact gap, declare fingerprint: per peer in the compose file (verified before the cluster is contacted; a mismatch is always fatal) or pin one host with arctic cluster trust. Read a host's fingerprint with arctic-agent fingerprint or from the agent's boot log. A corrupt trust store now fails closed with a clear error naming the file. See TLS and trust.
• Full security audit trail: Every authorization denial emits a WARN authz_denied event and every committed configuration change an INFO config_change event, complete across direct, forwarded, and gossip write paths. Steady-state heartbeats emit nothing, so the stream carries changes, not noise.
• Account lockout and an expiring recovery token: Five consecutive failed attempts lock a credential (or a recovery-token source IP) for 15 minutes, and the recovery token rotates every 24 hours instead of living until the next restart.
• Scoped, capped credentials with real revocation: Credential creation requires explicit scopes, grants are capped by the creator's scopes, and revoking a credential invalidates its outstanding tokens within about 30 seconds. Bootstrap leaves a least-privilege working credential in the CLI config and the admin pair in a break-glass file.
• Database files are no longer world-readable: arctic.db and its sidecars are enforced to mode 0600 on every open; an upgrade restart tightens files left loose by earlier versions.

Networking and data plane

• Device selection for multi-NIC hosts: The three device pins (routes[].device, interface.parent_dev, uplink_dev) are host-local operator intent, are not gossiped, and can be changed on a live service with arctic services update or a compose re-apply (removing the field clears the pin). Mixed-version caveat: a not-yet-upgraded source peer ignores the new fields on forwarded updates, so re-apply after completing the fleet upgrade. See Uplinks.
• Per-service MPTCP opt-in: subflows sets the requested subflow count (0 means the default of 2, maximum 8). Both knobs are source-peer-local dial settings and are rejected with KCP transport. The fallback to plain TCP is silent, so if throughput looks single-path, check kernel MPTCP support on both peers.
• Hardened systemd unit: agent install now emits a confined unit: still root, but with a reduced capability set and extensive systemd sandboxing (systemd-analyze security moves from 9.6 to 2.3). Upgrades preserve an existing unit; run agent install --force to adopt the hardened one. See Hardening.

Fixes

• Firewall state no longer accumulates: The firewall ruleset no longer grows on every state change, and stale routes and rules left behind by peer-membership and gateway changes are cleaned up automatically.
• TProxy: Two services targeting the same peer no longer conflict in a way that could wedge the proxy on a config reload; traffic now leaves by the interface the target was actually reached on, fixing a silent blackhole when a target routed over a non-default interface; a tunnel connection severed by a peer restart is retried instead of failing the stream; and uplink_dev now applies to the KCP transport as well.
• IP tunnel: Replies for client traffic forwarded through a target peer now route back into the tunnel correctly, and a startup race that could crash the tunnel engine is fixed.
• False peer drift is gone: compose apply and diff no longer report out-of-band drift against an unchanged cluster; a remaining peer drift report signals a real identity change. The CLI's local state file moved to a new format and migrates itself on load; an older CLI refuses a migrated file with a clear error instead of misreading it.
• Service recreation: Recreating a deleted service against a different peer now mints a fresh service ID and replicates cleanly across the cluster, instead of leaving the recreated service broken on other peers. Same-binding recreates keep their device pins, VLAN tag, and MAC mode.
• Compose backup edge cases: Same-second snapshots no longer overwrite an operator's rollback point, and a bootstrap run no longer presents its own pre-apply state as a prior backup.

CLI and UX

• Apply asks before touching anything: The confirmation prompt and --dry-run now resolve before any mutation. Fresh clusters get a full plan before a credential is even minted, and --dry-run on a fresh cluster no longer bootstraps it for real. One behavior change: a no-op apply no longer re-seeds peer endpoint lists; arctic cluster sync forces that.
• Rollback points: Every non-bootstrap apply captures a state-plus-config snapshot pair into .arctic/backup/ before executing. arctic state backups lists them and arctic compose rollback restores one through the standard apply pipeline, with its own plan preview and confirmation. See Backup and restore.
• Rebuilt-peer guard and arctic peers replace: Apply aborts before any mutation when an agent behind a member's endpoint has come back as a new identity, naming the stale member; peers replace verifies the rebuild, shows the blast radius, then swaps the member and re-applies in one step.
• Break-glass credential written to a file: Bootstrap writes the admin pair to break-glass.<cluster-id>.json (mode 0600) next to the CLI config as well as printing it once, so scripted runs no longer lose the cluster's only standing admin credential. --credentials-file none restores the old print-only posture.
• Project-scope target resolution: Commands run from a directory containing cluster.yaml target that cluster automatically, and the global --client-id/--client-secret flags now work for every command. A compose file plus ARCTIC_CLIENT_ID/ARCTIC_CLIENT_SECRET is a complete CI setup.
• Prunable services are surfaced: Without --prune, apply now lists the services that exist in the cluster but not in the config instead of reporting "No changes required", and includes them in the JSON result.
• Tidier local state: compose diff no longer creates .arctic/ on a fresh working directory, and apply writes a catch-all .gitignore inside .arctic/ so the cache cannot be committed by accident.

CLI patches (2026-07-22)

The v1.4.2 CLI binaries were re-issued on 2026-07-22 with the four fixes below. All four are client-side only: the agent binaries and the database schema are unchanged, and the patched CLI works with the original 2026-07-17 agents. The version string is the same; the quickest way to recognize a patched CLI is that arctic help lists the database command.

• Scheme-less compose endpoints default to https: A bare host or host:port peer endpoint in cluster.yaml is now read as https://, matching the agent's TLS-on default. This fixes the known issue where the first bootstrap of a fresh cluster failed with a connection-reset error unless every endpoint carried an explicit https:// scheme. Explicit schemes are still honored exactly as written. A deliberately plaintext cluster (PEER_TLS_ENABLED=false on every agent, or a TLS-terminating proxy in front of the API) must now declare explicit http:// endpoints; apply and diff print a note naming any peer that needs it, and compose export writes explicit http:// endpoints for a cluster reached over plaintext so exports still round-trip.
• A failed apply no longer strands a fresh cluster without credentials: When compose apply bootstraps a cluster and a later phase fails (a peer join, a service create), the cluster and its admin credential are now saved to the CLI config before the error is returned, so the next run can authenticate and retry. Previously the retry demanded credentials that were never saved and failed with 401, leaving the break-glass file as the only recovery. A later successful apply replaces the saved entry with the usual least-privilege working credential.
• Global boolean flags before the command work: Forms such as arctic --yes compose apply cluster.yaml and arctic -j health no longer fail with unknown command.
• arctic help lists every command: The database command and compose rollback were missing from the root help listing; their own help pages and behavior were unaffected.

1.4.1

A maintenance release that closes the last gap in the 1.4.0 "no host dependencies" promise and fixes several cluster and access issues. In 1.4.0 the agent programmed the kernel directly, but the QoS traffic shaper still shelled out to tc, ip, and modprobe. On a host hardened per the 1.4.0 guidance (those packages removed), enabling QoS would fail to start. 1.4.1 moves the QoS data plane onto the kernel directly as well, so the guarantee now holds whether or not QoS is enabled.

This release also makes the KCP tunnel transport selectable and extends bandwidth shaping to it, and fixes break-glass recovery, peer revocation, and a handful of startup and multi-homed networking issues.

Highlights

• No host dependencies, now including QoS: The traffic shaper now programs the kernel directly, so enabling QoS no longer requires iproute2 or the tc binary on the host. The running agent executes no external commands on its data path.
• KCP transport works again: A service set to transport_type: kcp once more uses the KCP (UDP-based) tunnel. A 1.4.0 regression silently ran these services over TCP instead.
• QoS shaping reaches KCP: Per-service bandwidth_limit_mbps now applies to KCP tunnels in both directions, not just TCP.
• Recovery and revocation fixes: The break-glass recovery token can now reliably mint credentials to restore access to a locked cluster, peer revocations converge reliably across the cluster, and a revoked peer can no longer silently rejoin.

Upgrading to 1.4.1

1.4.1 is a drop-in upgrade from 1.4.0. There are no configuration, schema, or API changes.

• You can remove iproute2 and tc now: If you kept these installed on a QoS-enabled host because of the 1.4.0 gap, they are no longer needed. QoS still requires the kernel itself to provide the shaping features (the sch_cake, sch_ingress, act_mirred, and ifb modules, built in or loadable); 1.4.1 removes the userland binary requirement, not the kernel-feature requirement.
• Clearing a peer revocation is now per node: Re-add the peer on each node that revoked it rather than relying on gossip to carry the reactivation.

Networking and data plane

• QoS over the kernel directly: The traffic shaper is now programmed directly through the kernel instead of shelling out to tc, ip, and modprobe. A genuinely missing kernel feature surfaces as a clear error and disables QoS gracefully.
• Fixed: KCP transport silently fell back to TCP: KCP worked before 1.4.0, but consolidating the data plane into the single agent introduced a regression where a transport_type: kcp service would quietly fall back to TCP. In the cases where KCP did activate, timing issues could break the link. KCP services are now wired through to the KCP tunnel reliably.
• Bandwidth shaping for KCP: bandwidth_limit_mbps now caps KCP traffic in both directions.
• Fixed: bandwidth_limit_mbps: 0 is now truly unlimited: With QoS enabled, a link with no explicit limit was silently capped at a 10 Gbit default instead of running unshaped. Such links are now left unshaped end to end. To turn QoS off for a whole node, disable qos_enabled on the peer.
• Fixed: startup under boot ordering: An agent that started before its default route existed (for example before a DHCP lease came up) could bind outbound links to the wrong interface, or leave QoS disabled, until a restart. Interface and QoS state are now resolved on every reconcile pass and self-heal within the resync window without a restart.
• Fixed: KCP on multi-homed and shared-subnet hosts: On a host whose service interface shared a subnet with the physical NIC, KCP tunnels could stall because replies went out from the wrong source address. The KCP listener now replies from the exact address each request arrived on, so KCP works on any multi-homed host.

Cluster and access fixes

• Recovery token can mint credentials: Presenting the break-glass recovery token previously unreliable when creating a credential, is now fixed.
• Peer revocations converge across the cluster: A peer revoked on one node could stay active on nodes that knew it only through gossip. Revocations now ride normal gossip and converge everywhere.
• Revoked peers cannot silently rejoin: A revoked peer that kept running could previously out-version its revocation and reactivate itself through gossip. Only an explicit operator re-add can now bring a revoked peer back.

1.4.0

This is a large release. The headline change is that Arctic is now a single agent binary: the separate proxy and tunnel services that earlier versions ran alongside the agent are now built into it. There is one binary to install and manage instead of three, and a target host no longer needs any extra packages or dependencies. It just has to be a machine running a Linux Kernel.

Beyond the consolidation, 1.4.0 brings tighter controls for locked-down deployments, safer cluster operations, and broader QoS tuning, plus fixes for a TCP throughput regression and a tunnel stall on poor links. The highlights and full details are below.

Highlights

• One binary: Arctic ships as a single agent. The binary dropped from roughly 71 MB to roughly 23 MB.
• No host dependencies: The agent runs no external commands and programs the kernel directly, so a host only needs to be a Linux machine (no NetworkManager, iproute2, nftables, or tc to install). This matters most for locked-down and embedded deployments.
• Locked-down operations: Expose a single node to operators and keep the rest reachable only from inside the cluster, with a break-glass recovery token to regain access if a cluster gets locked. See Access recovery and Hardening.
• Safer cluster changes: A cluster-wide apply lock prevents two operators applying at once, and version pinning keeps every peer on a compatible agent version.
• QoS for tough links: Fixed or default RTT, optional latency probing, and a memory cap for multi-gigabit links.

Upgrading to 1.4.0

1.4.0 changes how the agent is packaged and run. The upgrade is automatic, but review these notes first. See the Upgrades guide for the full procedure.

• Legacy services are removed automatically: On upgrade the installer stops, disables, and removes the old TProxy and IP-tunnel system units, their binaries, and their config files. The cleanup is safe to run more than once.
• No host packages required: Because the agent talks to the kernel directly, you no longer need NetworkManager, iproute2, nftables, or tc installed. They can be removed if nothing else on the host uses them.
• Peer address is deprecated: Use the new endpoints list instead. The single address field still works but produces a lint warning.
• Database migrations run on first start: The agent migrates its database automatically the first time 1.4.0 starts. Take a backup beforehand as usual. See Backup and restore.

Single agent binary

The data planes now run inside the agent process. The agent supervises them and restarts them if they fail. The firewall is now programmed directly through the kernel rather than by shelling out to command-line tools, so the agent does not depend on NetworkManager, iproute2, nftables, or tc being installed. On older kernels that lack nf_tables, the agent falls back to a legacy backend automatically (IPv4 only) and logs a warning so you can see it.

Locked-down access and recovery

• Internal API mode: Set a peer's api_access to internal so it refuses operator-facing API calls while still talking to other peers. This lets you expose a single node to operators and keep the rest reachable only from inside the cluster.
• Manage internal peers through the server: An operator does not lose control of an internal-only peer: a reachable, full-access peer transparently forwards operator requests to it over the cluster, so one front-door node can manage the whole cluster.
• Break-glass recovery token: At boot the agent writes a read-only recovery token to disk (mode 0600). Presenting it grants admin access on any request that carries it (until the next agent restart), even to a node in internal API mode, and every use is logged. Use it to regain control when a cluster is locked or an operator loses access. See Access recovery.
• Hardening guide: New documentation walks through locking down a Linux host that runs the agent. See Hardening.

Cluster management

• Version pinning: A compose requires.agent constraint forces every peer to satisfy a version range before an apply proceeds, so you avoid running mismatched features across a cluster. See Compose.
• Dedicated server role: Mark a peer as a server and name it in a server block as the node that is reliably reachable across the cluster.
• Fallback endpoints: A peer can list several endpoints (for example https://west.internal and 10.0.1.10:8080) that peers and operators try in order, so reachability can follow the client's network and internal DNS.
• Cluster apply lock: arctic compose apply now takes a cluster-wide lock so two operators cannot apply at once. A blocked apply reports the conflict, and a crashed operator's lock expires on its own. arctic compose diff warns when a lock is held but does not block.

Networking

• VLAN tagging: Set vlan_id on an interface to put a service's tunnel interface on an 802.1Q VLAN.
• Stable interface MAC addresses: Tunnel interfaces now default to a deterministic MAC derived from the cluster, peer, and service names, so the MAC stays the same across reboots without you pinning it by hand. You can still set an explicit MAC.
• Fixed: tunnel stall on poor links: On a very unreliable link, dropped handshakes and health checks could leave the IP-tunnel receiver wedged over time, eventually stalling all tunnel traffic until the service was restarted. The receiver now recovers on its own and no longer blocks.

See Configuration reference for the full set of interface options.

Performance and QoS

• Faster TCP transfer: A regression that forced TCP data through a slow user-space copy was fixed.
• Non-disruptive service updates: Changing one service now rebuilds only that service; the other services on the host keep running and their active connections survive.
• Default RTT: Set default_rtt_ms to seed the traffic shaper's round-trip time instead of relying on the built-in default.
• Disable RTT probing: Set disable_auto_rtt to fix the shaper to the configured RTT and stop probing, for volatile links where measured RTT swings unpredictably.
• QoS memory cap: Set memlimit_cap_mb to bound the memory the shaper uses. This matters on multi-gigabit links, where a 10 Gbit path can otherwise size its queues into several gigabytes of kernel memory and exhaust a memory-constrained host such as an embedded or field device.

See Configuration reference for the QoS options.

CLI and tooling

• compose diff: A new command shows the difference between your config file and the live cluster before you apply.
• Local state and history: The CLI keeps a working-directory cache in .arctic/ that tracks what was last applied (for accurate diffs), serializes concurrent runs on the same machine with a local lock, and keeps timestamped backups of previous applied states. Use arctic state unlock to clear a stuck lock.
• Stronger validation: Compose validation catches more mistakes up front (endpoint and MAC formatting, VLAN ranges, server-block and routing rules), and the same checks run in the Configuration Builder so the web tool and the CLI agree.
• Machine-readable output: Add --json to commands for stable output you can script against.
• MCP server: arctic mcp starts a local server that lets AI assistants such as Claude and ChatGPT drive the CLI against your cluster. Read operations are available by default; making changes requires opting in with --allow-write. See the CLI reference.
• Rebuilt configuration builder: The Configuration Builder was rebuilt. It now validates with the same code the CLI uses, compiled to run in the browser, so the web tool and the CLI won't drift.