Tillered Docs
Cluster Operations

Service management

How to create, configure, and manage services and routes in Arctic

Prefer compose for managing services

Services and routes are part of the compose configuration. For anything you keep around - defining services, adjusting routes, setting bandwidth limits - declare it in your cluster.yaml and run arctic compose apply. That keeps the cluster's desired state in version control and reviewable, and is the recommended way to manage Arctic. The commands on this page are best for inspecting state or making a quick one-off change.

Create a service

This section shows you how to create a service that routes traffic between two Arctic agents.

Before you start

Ensure you have:

  • At least two connected peers in your cluster
  • The target peer ID (the peer that will receive traffic)
  • Appropriate permissions (services.write scope)

Basic service creation

1. Find the target peer ID

List peers to find the ID of the destination peer:

arctic peers list

2. Create the service

Create a basic TCP service:

arctic services create --target-peer TARGET_PEER_ID
curl -X POST http://AGENT_IP:8080/v1/services \
  -H "Authorization: Bearer $TOKEN" \
  -H "Content-Type: application/json" \
  -d '{"target_peer_id": "TARGET_PEER_ID", "transport_type": "tcp"}'

target_peer_id and transport_type are required. The source peer defaults to the local peer when omitted.

This creates a service from the local peer to the target peer using TCP transport.

Advanced options

With MACVLAN interface

Create a service with a dedicated network interface for traffic isolation:

arctic services create \
  --target-peer TARGET_PEER_ID \
  --requires-interface
curl -X POST http://AGENT_IP:8080/v1/services \
  -H "Authorization: Bearer $TOKEN" \
  -H "Content-Type: application/json" \
  -d '{
    "target_peer_id": "TARGET_PEER_ID",
    "transport_type": "tcp",
    "requires_interface": true
  }'

With specific IP addresses

Request a specific IPv4 CIDR for the MACVLAN interface (omit to use DHCP):

arctic services create \
  --target-peer TARGET_PEER_ID \
  --requires-interface \
  --desired-ipv4 192.168.100.10/24
curl -X POST http://AGENT_IP:8080/v1/services \
  -H "Authorization: Bearer $TOKEN" \
  -H "Content-Type: application/json" \
  -d '{
    "target_peer_id": "TARGET_PEER_ID",
    "transport_type": "tcp",
    "requires_interface": true,
    "desired_interface_ipv4": "192.168.100.10/24"
  }'

With bandwidth limit

Apply a QoS bandwidth limit (in Mbps):

arctic services create \
  --target-peer TARGET_PEER_ID \
  --bandwidth-limit 1000
curl -X POST http://AGENT_IP:8080/v1/services \
  -H "Authorization: Bearer $TOKEN" \
  -H "Content-Type: application/json" \
  -d '{
    "target_peer_id": "TARGET_PEER_ID",
    "transport_type": "tcp",
    "qos": {"bandwidth_limit_mbps": 1000}
  }'

The bandwidth limit is in Mbps. Set to 0 for unlimited. See Set bandwidth limits below for details.

With fully transparent mode

Enable transparent mode to preserve the original source IP address at the destination:

arctic services create \
  --target-peer TARGET_PEER_ID \
  --fully-transparent \
  --requires-interface
curl -X POST http://AGENT_IP:8080/v1/services \
  -H "Authorization: Bearer $TOKEN" \
  -H "Content-Type: application/json" \
  -d '{
    "target_peer_id": "TARGET_PEER_ID",
    "transport_type": "tcp",
    "fully_transparent": true,
    "requires_interface": true
  }'

This is useful when the destination needs to see the real client IP for logging, access control, or rate limiting. See Transparent Mode for details.

Transparent mode requirements

Transparent mode requires --requires-interface and only works with TCP traffic.

With KCP transport

Use KCP instead of TCP for the underlying transport:

arctic services create \
  --target-peer TARGET_PEER_ID \
  --transport kcp
curl -X POST http://AGENT_IP:8080/v1/services \
  -H "Authorization: Bearer $TOKEN" \
  -H "Content-Type: application/json" \
  -d '{
    "target_peer_id": "TARGET_PEER_ID",
    "transport_type": "kcp"
  }'

KCP recovers from packet loss without backing off the way TCP does, and it starts fast without a slow warm-up, so it suits lossy links and short or interactive flows. High latency makes the short-flow case stronger, but a clean, high-latency link carrying a bulk transfer still favours TCP. See Choosing a transport for the full reasoning.

From a specific source peer

By default, the source peer is the local agent. To create a service from a different peer:

arctic services create \
  --source-peer SOURCE_PEER_ID \
  --target-peer TARGET_PEER_ID
curl -X POST http://AGENT_IP:8080/v1/services \
  -H "Authorization: Bearer $TOKEN" \
  -H "Content-Type: application/json" \
  -d '{
    "source_peer_id": "SOURCE_PEER_ID",
    "target_peer_id": "TARGET_PEER_ID",
    "transport_type": "tcp"
  }'

Complete example

Create a fully-configured service:

arctic services create \
  --target-peer peer_01HXYZDEF789... \
  --transport tcp \
  --requires-interface \
  --desired-ipv4 192.168.100.10/24 \
  --bandwidth-limit 1000
curl -X POST http://AGENT_IP:8080/v1/services \
  -H "Authorization: Bearer $TOKEN" \
  -H "Content-Type: application/json" \
  -d '{
    "target_peer_id": "peer_01HXYZDEF789...",
    "transport_type": "tcp",
    "requires_interface": true,
    "desired_interface_ipv4": "192.168.100.10/24",
    "qos": {"bandwidth_limit_mbps": 1000}
  }'

After creating a service

A service alone does not route traffic. You must add routes to specify which traffic should use the service. See Configure routes below for instructions on adding routing rules.

Troubleshooting

Target peer not found

If the target peer ID is not found:

  1. Verify the peer exists: arctic peers list
  2. Ensure the peer has completed handshake (not just discovered)

Service limit exceeded

If you receive a service limit error:

  1. Check your license: arctic license status
  2. Delete unused services or upgrade your license

Interface creation failed

If the MACVLAN interface fails to create:

  1. Check agent logs: journalctl -u arctic | grep netmgr
  2. Verify the host has a suitable parent interface
  3. Ensure the agent has root privileges

Configure routes

This section shows you how to add, update, and manage routing rules for Arctic services. Routes determine which traffic flows through a service based on source and destination CIDR blocks.

Before you start

Ensure you have:

  • An existing service (see Create a service above)
  • The service ID
  • Knowledge of the networks you want to route

Routes live on the source peer of a service. Route IDs are numeric.

Add a route

A route needs a --priority greater than 0 and at least one of --source-cidr or --dest-cidr.

Basic route

Add a route matching specific source and destination networks:

arctic routes add --service SERVICE_ID \
  --source-cidr 10.0.0.0/8 \
  --dest-cidr 192.168.100.0/24 \
  --priority 100
curl -X POST http://AGENT_IP:8080/v1/services/SERVICE_ID/routes \
  -H "Authorization: Bearer $TOKEN" \
  -H "Content-Type: application/json" \
  -d '{
    "routes": [{
      "source_cidr": "10.0.0.0/8",
      "dest_cidr": "192.168.100.0/24",
      "priority": 100
    }]
  }'

The request body carries a routes array, so you can add several routes in one call. The response returns the new route_ids and a count.

Destination-only route

Route all traffic to a specific destination regardless of source:

arctic routes add --service SERVICE_ID \
  --dest-cidr 192.168.100.0/24 \
  --priority 100
curl -X POST http://AGENT_IP:8080/v1/services/SERVICE_ID/routes \
  -H "Authorization: Bearer $TOKEN" \
  -H "Content-Type: application/json" \
  -d '{
    "routes": [{
      "dest_cidr": "192.168.100.0/24",
      "priority": 100
    }]
  }'

Source-only route

Route all traffic from a specific source regardless of destination:

arctic routes add --service SERVICE_ID \
  --source-cidr 10.0.0.0/8 \
  --priority 100
curl -X POST http://AGENT_IP:8080/v1/services/SERVICE_ID/routes \
  -H "Authorization: Bearer $TOKEN" \
  -H "Content-Type: application/json" \
  -d '{
    "routes": [{
      "source_cidr": "10.0.0.0/8",
      "priority": 100
    }]
  }'

Understanding priority

Routes are matched by specificity first, then priority breaks ties.

Lower priority value wins

A LOWER priority value means a HIGHER-priority route. Priority 100 beats priority 200 when specificity is equal.

Specificity is evaluated in this order (most specific first):

  1. MACVLAN interface match
  2. Source CIDR + destination CIDR match
  3. Source-only CIDR match
  4. Destination-only CIDR match

When two routes have the same specificity, the one with the lower priority value is chosen.

Example ordering (most preferred first):

PrioritySource CIDRDest CIDRReason
10010.1.0.0/16192.168.100.0/24More specific source (src+dest match)
20010.0.0.0/8192.168.100.0/24Less specific source
100(none)192.168.0.0/16Dest-only match, least specific

List routes

View all routes for a service:

arctic routes list --service SERVICE_ID
curl -X GET http://AGENT_IP:8080/v1/services/SERVICE_ID/routes \
  -H "Authorization: Bearer $TOKEN"

The response wraps the list under a routes array.

Update a route

Modify an existing route's priority or CIDR:

arctic routes update --service SERVICE_ID --route ROUTE_ID \
  --priority 50
curl -X PUT http://AGENT_IP:8080/v1/services/SERVICE_ID/routes/ROUTE_ID \
  -H "Authorization: Bearer $TOKEN" \
  -H "Content-Type: application/json" \
  -d '{
    "priority": 50,
    "source_cidr": "10.0.0.0/8",
    "dest_cidr": "192.168.100.0/24"
  }'

You can also update the CIDRs from the CLI:

arctic routes update --service SERVICE_ID --route ROUTE_ID \
  --source-cidr 10.1.0.0/16 \
  --dest-cidr 192.168.200.0/24

Delete a route

Remove a route from a service:

arctic routes delete --service SERVICE_ID --route ROUTE_ID

# Use --yes to skip confirmation:
arctic routes delete --service SERVICE_ID --route ROUTE_ID --yes
curl -X DELETE http://AGENT_IP:8080/v1/services/SERVICE_ID/routes/ROUTE_ID \
  -H "Authorization: Bearer $TOKEN"

A successful delete returns HTTP 204 with no body.

Common routing patterns

Route all traffic to a service

Route everything from one network to another:

arctic routes add --service SERVICE_ID \
  --source-cidr 10.0.0.0/8 \
  --dest-cidr 0.0.0.0/0 \
  --priority 100

Multiple routes for different subnets

Add routes for different destination subnets:

# Route to subnet A
arctic routes add --service SERVICE_ID \
  --dest-cidr 192.168.100.0/24 \
  --priority 100

# Route to subnet B
arctic routes add --service SERVICE_ID \
  --dest-cidr 192.168.200.0/24 \
  --priority 100

Override with a higher-priority route

Add a more specific route, then give it a lower priority value so it wins the tie:

# General route (lower priority: higher value)
arctic routes add --service SERVICE_ID \
  --source-cidr 10.0.0.0/8 \
  --dest-cidr 192.168.0.0/16 \
  --priority 200

# Specific override (higher priority: lower value)
arctic routes add --service SERVICE_ID \
  --source-cidr 10.1.0.0/16 \
  --dest-cidr 192.168.100.0/24 \
  --priority 100

The specific route is already preferred on specificity; the lower priority value reinforces the override when specificity is equal.

Troubleshooting

Routes not taking effect

If traffic is not being routed as expected:

  1. Trigger a config sync: arctic cluster sync
  2. Inspect the kernel ruleset: nft list table inet arctic

Invalid CIDR notation

Ensure CIDRs are valid:

  • Use slash notation: 10.0.0.0/8 not 10.0.0.0
  • The network address must match the mask: 10.0.0.0/8 not 10.1.2.3/8

Priority must be greater than zero

arctic routes add requires a --priority value greater than 0.

Route not found

If a route ID is not found:

  1. List routes to verify: arctic routes list --service SERVICE_ID
  2. Ensure you are using the correct service ID

Set bandwidth limits

This section shows you how to configure bandwidth limits on Arctic services for Quality of Service (QoS) control.

How limits apply to TCP and KCP

On a TCP service, bandwidth_limit_mbps drives the full traffic shaper on the in-process TProxy data plane (fair queueing and latency control). On a KCP service it acts as a bandwidth ceiling that caps the link in both directions, without the shaper's fair queueing. The other qos fields (RTT and memory tuning) apply to TCP shaping only.

Before you start

Ensure you have:

  • An existing service or the information to create one
  • Knowledge of the desired bandwidth limit in Mbps

Set bandwidth during service creation

Specify a bandwidth limit when creating a new service:

arctic services create \
  --target-peer TARGET_PEER_ID \
  --bandwidth-limit 1000
curl -X POST http://AGENT_IP:8080/v1/services \
  -H "Authorization: Bearer $TOKEN" \
  -H "Content-Type: application/json" \
  -d '{
    "target_peer_id": "TARGET_PEER_ID",
    "transport_type": "tcp",
    "qos": {"bandwidth_limit_mbps": 1000}
  }'

The value is in megabits per second (Mbps). 1000 means 1 Gbps.

Update bandwidth on an existing service

Change the bandwidth limit on an existing service:

arctic services update SERVICE_ID --bandwidth-limit 500
curl -X PUT http://AGENT_IP:8080/v1/services/SERVICE_ID \
  -H "Authorization: Bearer $TOKEN" \
  -H "Content-Type: application/json" \
  -d '{"bandwidth_limit_mbps": 500}'

Remove the bandwidth limit

Set the limit to 0 to remove the restriction (unlimited):

arctic services update SERVICE_ID --bandwidth-limit 0
curl -X PUT http://AGENT_IP:8080/v1/services/SERVICE_ID \
  -H "Authorization: Bearer $TOKEN" \
  -H "Content-Type: application/json" \
  -d '{"bandwidth_limit_mbps": 0}'

Verify current bandwidth

Check the current bandwidth setting:

arctic services get SERVICE_ID
curl -X GET http://AGENT_IP:8080/v1/services/SERVICE_ID \
  -H "Authorization: Bearer $TOKEN"

Look for the bandwidth_limit_mbps field in the output.

Common bandwidth values

Use caseBandwidthValue
Low priority100 Mbps100
Standard1 Gbps1000
High throughput10 Gbps10000
UnlimitedNo limit0

How bandwidth limiting works

For a TCP service Arctic shapes traffic on the TProxy data plane: it holds the traffic at the configured limit, shares capacity fairly between flows, and keeps latency low when the link is busy. For a KCP service the limit is enforced as a bandwidth ceiling in both directions rather than full shaping, so it caps total throughput but does not queue flows fairly or manage latency.

Bandwidth limits apply to traffic flowing through the service. They do not affect traffic that does not match the service's routes.

Troubleshooting

Bandwidth not being enforced

If traffic exceeds the configured limit:

  1. Confirm QoS is enabled for the target peer: arctic peers get PEER_ID
  2. Trigger a config sync: arctic cluster sync
  3. Check agent logs: journalctl -u arctic

Performance lower than expected

If throughput is below the configured limit:

  1. Check for network bottlenecks elsewhere in the path
  2. Verify the underlying network can support the desired bandwidth
  3. Consider TCP tuning for high-bandwidth scenarios

Delete a service

This section shows you how to delete an Arctic service and clean up its associated configuration.

Before you start

Understand that deleting a service:

  • Removes all routes associated with the service
  • Removes the MACVLAN interface (if one was created)
  • Updates firewall rules to stop routing traffic
  • Does not affect the source or target peers

Steps

1. Find the service ID

List services to find the ID:

arctic services list
curl -X GET http://AGENT_IP:8080/v1/services \
  -H "Authorization: Bearer $TOKEN"

The response wraps the list under a services array.

2. Delete the service

arctic services delete SERVICE_ID

# You will be prompted to confirm. Use --yes to skip confirmation:
arctic services delete SERVICE_ID --yes
curl -X DELETE http://AGENT_IP:8080/v1/services/SERVICE_ID \
  -H "Authorization: Bearer $TOKEN"

A successful delete returns HTTP 204 with no body.

3. Verify deletion

Confirm the service was removed:

arctic services list

The service should no longer appear in the list.

What happens when you delete

When a service is deleted:

  1. Routes removed: All CIDR-based routing rules are deleted.
  2. Interface cleaned up: The MACVLAN interface is removed from the host.
  3. Firewall updated: The agent re-commits the nftables ruleset so it no longer matches traffic for this service.
  4. Data plane updated: The in-process TProxy and IP-tunnel engines are reconfigured.

Configuration changes propagate automatically, typically within seconds.

Bulk deletion

To delete several services, filter the list to JSON and loop over the IDs. The --json output is a top-level array of service objects, each with an id field:

# Delete all services targeting a specific peer
for id in $(arctic services list --target-peer PEER_ID --json | jq -r '.[].id'); do
  arctic services delete "$id" --yes
done

Troubleshooting

Service not found

If the service ID is not found:

  1. Verify the service exists: arctic services list
  2. Check you are connected to the correct agent
  3. The service may have already been deleted

Interface not removed

If the MACVLAN interface persists after deletion:

  1. Check the network reconciler logs: journalctl -u arctic | grep netmgr
  2. Manually remove if needed: ip link delete INTERFACE_NAME
  3. Trigger a sync: arctic cluster sync

Routes still active

If traffic is still being routed after deletion:

  1. Trigger a config sync: arctic cluster sync
  2. Inspect the kernel ruleset: nft list table inet arctic
  3. Check agent logs for config updates: journalctl -u arctic

On this page