Tillered Docs
Maintenance

Hardening

The hardened default unit, running the agent as a non-root user, and the host posture around it

As of v1.4.2 the default installation is already confined: agent install writes a hardened systemd unit that keeps the agent as root but strips most of what root can normally do. This page describes what that default gives you, and the stricter step beyond it: a hand-built unit that runs the agent as a dedicated unprivileged user holding a single Linux capability.

The default hardened unit

The unit agent install writes still runs the agent as root (the data directory and recovery token are root-owned on a default install), but the process is sandboxed:

  • The capability bounding set is reduced to CAP_NET_ADMIN, CAP_NET_RAW, and CAP_NET_BIND_SERVICE, with NoNewPrivileges set.
  • ProtectSystem=strict makes the filesystem read-only apart from the data directory, /etc/arctic, and /proc/sys/net.
  • A closed device policy exposes only /dev/net/tun.
  • Kernel-surface protections and a @system-service syscall filter apply.
  • LimitNOFILE=1048576 grants the file-descriptor headroom the engine needs, since with CAP_SYS_RESOURCE dropped it can no longer raise its own limit.

systemd-analyze security arctic scores this unit 2.3 (OK); the pre-1.4.2 unconfined unit scored 9.6 (UNSAFE).

Changed in v1.4.2

Earlier releases installed an unconfined root unit, and upgrades preserve an existing (possibly customized) unit, so upgrading alone does not adopt the new one. Run sudo agent install --force on an existing host to adopt the hardened unit.

Beyond the default: dropping root

The agent does not actually need root; it needs a single Linux capability, CAP_NET_ADMIN, for everything it does to kernel network state. The rest of this guide replaces the bundled unit with a hand-built one that runs the agent as a dedicated unprivileged user holding only that capability, wrapped in the same systemd sandbox, so that a compromised agent can touch as little as possible.

You take over the lifecycle

A hand-built non-root unit is incompatible with the bundled lifecycle commands. Do not run agent install or agent upgrade on a host set up this way: both require root, agent install --force replaces your unit with the bundled root unit, and agent upgrade swaps the binary at the bundled path rather than the root-owned copy this guide installs, so your unit would keep running the old binary. Once you go down this path, you install, upgrade, and manage the service by hand. The steps for a manual upgrade are at the end of this page.

This is an advanced, operator-maintained configuration. It is not the path the installer takes and is not exercised by the project's release testing, so validate it in staging before you rely on it.

What the agent needs

The agent process has no root check; it reads and writes kernel network state through one capability:

  • CAP_NET_ADMIN is the one that matters, and on a host with nftables (every modern distribution) it is the only one required. It covers committing firewall rules over netlink, creating MACVLAN and TUN devices, managing ip rules and routes, the transparent-proxy socket option (IP_TRANSPARENT, which the kernel grants under CAP_NET_ADMIN), writing the network sysctls the agent manages, and the traffic-shaping queues used for QoS.
  • CAP_NET_RAW is needed only on hosts that fall back to the legacy xtables path because nftables is unavailable; that path opens a raw socket. Most hosts never use it. If yours does, add it as shown below.
  • No privileged ports are involved. The API (8080), TProxy (61000), and IP tunnel (51840) all bind above 1024, so CAP_NET_BIND_SERVICE is not required.

The in-process IP tunnel uses per-peer TUN devices rather than network namespaces, so CAP_SYS_ADMIN is not required either. That is what makes a non-root agent practical.

At runtime the agent writes to three kinds of location, and nothing else on disk:

  • Its data directory (/opt/tillered by default), holding the database and peer.key.
  • /etc/arctic, where it rotates the recovery token every 24 hours (see access recovery).
  • A handful of network sysctls under /proc/sys/net (IP forwarding and rp_filter), written directly to procfs rather than through the sysctl binary. The kernel permits these writes on the strength of CAP_NET_ADMIN, so the non-root agent can make them, but the path has to be writable in the unit's mount namespace (see ReadWritePaths below).

Everything else the agent does is kernel state over netlink, not files.

Manual setup

1. Create a service user

sudo groupadd --system arctic
sudo useradd --system --gid arctic --no-create-home \
  --shell /usr/sbin/nologin arctic

2. Install the binary

Download the agent binary and place it outside the data directory, owned by root, so the service user can neither replace the binary nor rename the directory that holds it:

sudo curl -fsSL https://release.tillered.com/arctic/latest/agent_amd64 \
  -o /usr/local/bin/arctic-agent
sudo chown root:root /usr/local/bin/arctic-agent
sudo chmod 0755 /usr/local/bin/arctic-agent

Use agent_arm64 on ARM hosts. Verify the binary signature before trusting it; see Installation.

3. Create and own the data directory

The service user owns the data directory. The recovery-token directory (/etc/arctic) is created for you by systemd in step 5 via ConfigurationDirectory, owned by the service user; you do not create it by hand.

sudo mkdir -p /opt/tillered
sudo chown arctic:arctic /opt/tillered
sudo chmod 0700 /opt/tillered

4. Kernel modules for QoS

QoS relies on the kernel's traffic-shaping modules. Under CAP_NET_ADMIN the kernel loads them on demand, so a typical host needs no action here. On hosts that block on-demand module loading, make sure the traffic-shaping modules your kernel uses are available at boot.

5. Write the systemd unit

Create /etc/systemd/system/arctic.service. This is the bundled unit's hardening set, adapted to run as the unprivileged arctic user: the capability is granted ambiently (a non-root process must be handed the capability, not just allowed to keep it), and the writable paths are held to the three the agent actually uses.

[Unit]
Description=Arctic Network Routing Agent
After=network-online.target
Wants=network-online.target

[Service]
Type=simple
User=arctic
Group=arctic
ExecStart=/usr/local/bin/arctic-agent

Restart=on-failure
RestartSec=5s
StandardOutput=journal
StandardError=journal
Environment=DATA_DIR=/opt/tillered
Environment=LOG_FORMAT=json

# Capabilities: hand the non-root process CAP_NET_ADMIN ambiently and bound
# the set to it. Add CAP_NET_RAW to BOTH lines only on hosts without nftables
# (the xtables fallback opens a raw socket).
AmbientCapabilities=CAP_NET_ADMIN
CapabilityBoundingSet=CAP_NET_ADMIN
NoNewPrivileges=yes
LimitNOFILE=1048576

# Filesystem: read-only OS, write access only to the data dir, the
# recovery-token dir, and net sysctls (forwarding / rp_filter). systemd
# creates /etc/arctic first (owned by the service user, mode 0700); the
# matching ReadWritePaths line is still required, because a
# ConfigurationDirectory stays read-only under ProtectSystem=strict.
ProtectSystem=strict
ProtectHome=yes
ReadWritePaths=/opt/tillered
ReadWritePaths=/proc/sys/net
ConfigurationDirectory=arctic
ConfigurationDirectoryMode=0700
ReadWritePaths=/etc/arctic
PrivateTmp=yes
UMask=0077

# Devices: only the TUN clone device (iptun engine).
DevicePolicy=closed
DeviceAllow=/dev/net/tun rw

# Kernel surface
ProtectKernelTunables=yes
ProtectKernelModules=yes
ProtectKernelLogs=yes
ProtectClock=yes
ProtectControlGroups=yes
ProtectHostname=yes
ProtectProc=invisible

# Syscall and misc restrictions
RestrictAddressFamilies=AF_UNIX AF_INET AF_INET6 AF_NETLINK
RestrictNamespaces=yes
RestrictRealtime=yes
RestrictSUIDSGID=yes
LockPersonality=yes
MemoryDenyWriteExecute=yes
RemoveIPC=yes
SystemCallFilter=@system-service
SystemCallErrorNumber=EPERM
SystemCallArchitectures=native

[Install]
WantedBy=multi-user.target

On hosts without nftables that need the xtables fallback, add CAP_NET_RAW to both AmbientCapabilities and CapabilityBoundingSet.

6. Enable and start

sudo systemctl daemon-reload
sudo systemctl enable --now arctic

7. Bootstrap

Bootstrapping is unchanged; it is an API call to the running agent. The agent serves HTTPS with a self-signed certificate by default, so use an https URL; the CLI pins the certificate on first contact (trust on first use), so no --insecure is needed:

arctic bootstrap --url https://localhost:8080 \
  --license-file license.json --name node-a

Verify

Confirm the agent answers and is running as the unprivileged user (the agent serves HTTPS with a self-signed certificate, so -k skips local cert verification):

curl -sk https://localhost:8080/livez

systemctl show arctic --property User --property MainPID
ps -o user= -p "$(systemctl show arctic --property MainPID --value)"

The ps output should read arctic, not root. Confirm the process holds CAP_NET_ADMIN and nothing more; the effective set should be a single bit:

grep Cap /proc/"$(systemctl show arctic --property MainPID --value)"/status
capsh --decode=0000000000001000   # -> cap_net_admin

Check that systemd created the recovery-token directory and that the agent wrote the token into it under the service user:

sudo ls -ld /etc/arctic
sudo ls -l /etc/arctic/recovery.token

If the token is missing, the agent could not write /etc/arctic; recovery is disabled but the agent otherwise runs normally. Re-check that the unit still carries the ConfigurationDirectory=arctic and ReadWritePaths=/etc/arctic lines. Finally, confirm the managed sysctls actually took, which proves /proc/sys/net is writable in the sandbox:

cat /proc/sys/net/ipv4/ip_forward   # -> 1

Converting an existing root install

If a host already runs the default root install (from agent install) and you want to move it to non-root without a clean rebuild, the conversion hands the existing state to a new service user. Do it in one maintenance window: the agent must be stopped while its data directory changes owner.

# 1. Stop the agent.
sudo systemctl stop arctic

# 2. Create the service user (as in Manual setup, step 1).
sudo groupadd --system arctic
sudo useradd --system --gid arctic --no-create-home \
  --shell /usr/sbin/nologin arctic

# 3. Relocate the binary out of the data directory, root-owned.
sudo cp /opt/tillered/bin/arctic-agent /usr/local/bin/arctic-agent
sudo chown root:root /usr/local/bin/arctic-agent
sudo chmod 0755 /usr/local/bin/arctic-agent
# On SELinux hosts, label the relocated binary so systemd can exec it:
# sudo restorecon -v /usr/local/bin/arctic-agent

# 4. Hand the data directory (database, peer.key) to the service user.
sudo chown -R arctic:arctic /opt/tillered
sudo chmod 0700 /opt/tillered

# 5. Install the non-root unit from Manual setup, step 5, then:
sudo systemctl daemon-reload
sudo systemctl start arctic

Verify as above; the process should now run as arctic. The old /opt/tillered/bin/arctic-agent is now owned by the service user and unused (the unit points at /usr/local/bin); remove it once the conversion is confirmed.

Rollback

The only destructive step is the data-directory chown. To revert, restore the previous root unit and sudo systemctl restart arctic. Root reads the now-arctic-owned database and key without a second chown, so no further change is needed to go back.

The bundled installer can do most of this for you with sudo agent install --non-root --force, which creates the user, relocates the binary, chowns the data directory, and writes the non-root unit. It does not stop or restart a running service, so stop arctic before running it and restart afterward, exactly as in the manual steps above.

Upgrading by hand

Because you cannot use agent upgrade, upgrade by replacing the binary and restarting. Your unit file is left untouched:

sudo systemctl stop arctic
sudo curl -fsSL https://release.tillered.com/arctic/latest/agent_amd64 \
  -o /usr/local/bin/arctic-agent
sudo chown root:root /usr/local/bin/arctic-agent
sudo chmod 0755 /usr/local/bin/arctic-agent
sudo systemctl start arctic

Verify the new version afterwards:

journalctl -u arctic --since "2 minutes ago"
arctic license status

For a multi-peer cluster, check the release notes for the required order before you start: the usual pattern is one peer at a time, but some releases (v1.4.2 among them) require every node to upgrade together. See Upgrades.

Caveats

  • /dev/net/tun must exist and be accessible. On most distributions it is world-accessible (crw-rw-rw-). If it is missing, load the tun module.
  • /proc/sys/net must stay in ReadWritePaths. The agent writes forwarding and rp_filter sysctls directly to procfs on every reconcile. Drop that line and the writes fail silently under ProtectSystem=strict; routing then breaks in ways that are hard to trace back to the unit.
  • Recovery depends on /etc/arctic being writable by the service user. If the ConfigurationDirectory or ReadWritePaths=/etc/arctic line is removed, break-glass recovery is unavailable on that host.
  • ProtectSystem=strict makes the filesystem read-only apart from ReadWritePaths. If you move the data directory, update the DATA_DIR environment line and the matching ReadWritePaths entry together.
  • Tighten further only with care. The unit above is already close to the minimum surface: netlink, packet, and inet socket families, one capability, three writable paths. Further restrictions such as a stricter SystemCallFilter allowlist can help, but test each change in staging, because the agent's netlink and traffic-shaping paths exercise a wide syscall surface.
  • You own upgrades and unit changes. Nothing in the agent will rewrite or update this unit for you, which is the point, but it also means security updates to the recommended unit are yours to apply.

Keeping a host firewall

This guide hardens the agent process; it does not add a host firewall. The default setup removes firewalld or ufw so the agent's own kernel rules are the only thing filtering traffic. If your hardening posture requires keeping a host firewall, you take on managing its rules alongside the agent. On firewalld hosts this needs more than opening the three ports: the agent's forwarding interfaces (tpt-*, svc_*) must be placed in the LAN interface's zone or forwarded service traffic is silently rejected. See Keeping firewalld for the full explanation and commands.

File modes and the audit trail

Two smaller v1.4.2 changes round out the host posture. The database is no longer world-readable: arctic.db and its WAL/SHM sidecars are enforced to mode 0600 on every open, matching peer.key, and an upgrade restart tightens files left loose by earlier versions. And every authorization denial and committed configuration change now emits a structured audit event; see audit events for what the trail contains and what to watch.

See also

On this page