Tillered Docs
Cluster Operations

Service management

How to create, configure, and manage services and routes in Arctic

Create a service

This section shows you how to create a service that routes traffic between two Arctic agents.

Before you start

Ensure you have:

  • At least two connected peers in your cluster
  • The target peer ID (the peer that will receive traffic)
  • Appropriate permissions (services.write scope)

Basic service creation

1. Find the target peer ID

List peers to find the ID of the destination peer:

arctic peers list

2. Create the service

Create a basic TCP service:

arctic services create --target-peer TARGET_PEER_ID
curl -X POST http://AGENT_IP:8080/v1/services \
  -H "Authorization: Bearer $TOKEN" \
  -H "Content-Type: application/json" \
  -d '{"target_peer_id": "TARGET_PEER_ID"}'

This creates a service from the local peer to the target peer using TCP transport.

Advanced options

With MACVLAN interface

Create a service with a dedicated network interface for traffic isolation:

arctic services create \
  --target-peer TARGET_PEER_ID \
  --requires-interface
curl -X POST http://AGENT_IP:8080/v1/services \
  -H "Authorization: Bearer $TOKEN" \
  -H "Content-Type: application/json" \
  -d '{
    "target_peer_id": "TARGET_PEER_ID",
    "requires_interface": true
  }'

With specific IP addresses

Request specific IP addresses for the MACVLAN interface:

arctic services create \
  --target-peer TARGET_PEER_ID \
  --requires-interface \
  --desired-ipv4 192.168.100.10/24
curl -X POST http://AGENT_IP:8080/v1/services \
  -H "Authorization: Bearer $TOKEN" \
  -H "Content-Type: application/json" \
  -d '{
    "target_peer_id": "TARGET_PEER_ID",
    "requires_interface": true,
    "desired_ipv4_cidr": "192.168.100.10/24"
  }'

With bandwidth limit

Apply QoS bandwidth limiting:

arctic services create \
  --target-peer TARGET_PEER_ID \
  --bandwidth-limit 1000
curl -X POST http://AGENT_IP:8080/v1/services \
  -H "Authorization: Bearer $TOKEN" \
  -H "Content-Type: application/json" \
  -d '{
    "target_peer_id": "TARGET_PEER_ID",
    "bandwidth_limit_mbps": 1000
  }'

The bandwidth limit is in Mbps. Set to 0 for unlimited.

With fully transparent mode

Enable Transparent Mode to preserve the original source IP address at the destination:

arctic services create \
  --target-peer TARGET_PEER_ID \
  --fully-transparent \
  --requires-interface
curl -X POST http://AGENT_IP:8080/v1/services \
  -H "Authorization: Bearer $TOKEN" \
  -H "Content-Type: application/json" \
  -d '{
    "target_peer_id": "TARGET_PEER_ID",
    "fully_transparent": true,
    "requires_interface": true
  }'

This is useful when the destination needs to see the real client IP for logging, access control, or rate limiting. See Transparent Mode for details.

Note: Transparent Mode requires --requires-interface and only works with TCP traffic.

With KCP transport

Use KCP instead of TCP for the underlying transport:

arctic services create \
  --target-peer TARGET_PEER_ID \
  --transport kcp
curl -X POST http://AGENT_IP:8080/v1/services \
  -H "Authorization: Bearer $TOKEN" \
  -H "Content-Type: application/json" \
  -d '{
    "target_peer_id": "TARGET_PEER_ID",
    "transport_type": "kcp"
  }'

KCP provides better performance over high-latency or lossy networks.

From a specific source peer

By default, the source peer is the local agent. To create a service from a different peer:

arctic services create \
  --source-peer SOURCE_PEER_ID \
  --target-peer TARGET_PEER_ID
curl -X POST http://AGENT_IP:8080/v1/services \
  -H "Authorization: Bearer $TOKEN" \
  -H "Content-Type: application/json" \
  -d '{
    "source_peer_id": "SOURCE_PEER_ID",
    "target_peer_id": "TARGET_PEER_ID"
  }'

Complete example

Create a fully-configured service:

arctic services create \
  --target-peer 01HXYZDEF789... \
  --transport tcp \
  --requires-interface \
  --desired-ipv4 192.168.100.10/24 \
  --bandwidth-limit 1000
curl -X POST http://AGENT_IP:8080/v1/services \
  -H "Authorization: Bearer $TOKEN" \
  -H "Content-Type: application/json" \
  -d '{
    "target_peer_id": "01HXYZDEF789...",
    "transport_type": "tcp",
    "requires_interface": true,
    "desired_ipv4_cidr": "192.168.100.10/24",
    "bandwidth_limit_mbps": 1000
  }'

After creating a service

A service alone does not route traffic. You must add routes to specify which traffic should use the service. See Configure routes below for instructions on adding routing rules.

Troubleshooting

Target Peer Not Found

If the target peer ID is not found:

  1. Verify the peer exists: arctic peers list
  2. Ensure the peer has completed handshake (not just discovered)

Service Limit Exceeded

If you receive a service limit error:

  1. Check your license: arctic license show
  2. Delete unused services or upgrade your license

Interface Creation Failed

If the MACVLAN interface fails to create:

  1. Check agent logs: journalctl -u arctic | grep netmgr
  2. Verify the host has a suitable parent interface
  3. Ensure the agent has root privileges

Configure routes

This section shows you how to add, update, and manage routing rules for Arctic services. Routes determine which traffic flows through a service based on source and destination CIDR blocks.

Before you start

Ensure you have:

  • An existing service (see Create a service above)
  • The service ID
  • Knowledge of the networks you want to route

Add a route

Basic route

Add a route matching specific source and destination networks:

arctic routes add --service SERVICE_ID \
  --source-cidr 10.0.0.0/8 \
  --dest-cidr 192.168.100.0/24 \
  --priority 100
curl -X POST http://AGENT_IP:8080/v1/services/SERVICE_ID/routes \
  -H "Authorization: Bearer $TOKEN" \
  -H "Content-Type: application/json" \
  -d '{
    "routes": [{
      "source_cidr": "10.0.0.0/8",
      "dest_cidr": "192.168.100.0/24",
      "priority": 100
    }]
  }'

Destination-only route

Route all traffic to a specific destination regardless of source:

arctic routes add --service SERVICE_ID \
  --dest-cidr 192.168.100.0/24 \
  --priority 100
curl -X POST http://AGENT_IP:8080/v1/services/SERVICE_ID/routes \
  -H "Authorization: Bearer $TOKEN" \
  -H "Content-Type: application/json" \
  -d '{
    "routes": [{
      "dest_cidr": "192.168.100.0/24",
      "priority": 100
    }]
  }'

Source-only route

Route all traffic from a specific source regardless of destination:

arctic routes add --service SERVICE_ID \
  --source-cidr 10.0.0.0/8 \
  --priority 100
curl -X POST http://AGENT_IP:8080/v1/services/SERVICE_ID/routes \
  -H "Authorization: Bearer $TOKEN" \
  -H "Content-Type: application/json" \
  -d '{
    "routes": [{
      "source_cidr": "10.0.0.0/8",
      "priority": 100
    }]
  }'

Understanding priority

Routes are evaluated by specificity and priority:

  1. Most specific match wins: A route with /24 CIDR is more specific than /16
  2. Higher priority wins: When specificity is equal, higher priority value wins

Example ordering (most preferred first):

PrioritySource CIDRDest CIDRReason
20010.1.0.0/16192.168.100.0/24Most specific source
10010.0.0.0/8192.168.100.0/24Less specific source
10010.0.0.0/8192.168.0.0/16Less specific dest

List routes

View all routes for a service:

arctic routes list --service SERVICE_ID
curl -X GET http://AGENT_IP:8080/v1/services/SERVICE_ID/routes \
  -H "Authorization: Bearer $TOKEN"

Update a route

Modify an existing route's priority or CIDR:

arctic routes update --service SERVICE_ID --route ROUTE_ID \
  --priority 200
curl -X PUT http://AGENT_IP:8080/v1/services/SERVICE_ID/routes/ROUTE_ID \
  -H "Authorization: Bearer $TOKEN" \
  -H "Content-Type: application/json" \
  -d '{"priority": 200}'

You can also update the CIDRs:

arctic routes update --service SERVICE_ID --route ROUTE_ID \
  --source-cidr 10.1.0.0/16 \
  --dest-cidr 192.168.200.0/24

Delete a route

Remove a route from a service:

arctic routes delete --service SERVICE_ID --route ROUTE_ID

# Use --yes to skip confirmation:
arctic routes delete --service SERVICE_ID --route ROUTE_ID --yes
curl -X DELETE http://AGENT_IP:8080/v1/services/SERVICE_ID/routes/ROUTE_ID \
  -H "Authorization: Bearer $TOKEN"

Common routing patterns

Route all traffic to a service

Route everything from one network to another:

arctic routes add --service SERVICE_ID \
  --source-cidr 10.0.0.0/8 \
  --dest-cidr 0.0.0.0/0 \
  --priority 100

Multiple routes for different subnets

Add routes for different destination subnets:

# Route to subnet A
arctic routes add --service SERVICE_ID \
  --dest-cidr 192.168.100.0/24 \
  --priority 100

# Route to subnet B
arctic routes add --service SERVICE_ID \
  --dest-cidr 192.168.200.0/24 \
  --priority 100

Override with higher priority

Add a more specific route that overrides a general one:

# General route (lower priority)
arctic routes add --service SERVICE_ID \
  --source-cidr 10.0.0.0/8 \
  --dest-cidr 192.168.0.0/16 \
  --priority 100

# Specific override (higher priority)
arctic routes add --service SERVICE_ID \
  --source-cidr 10.1.0.0/16 \
  --dest-cidr 192.168.100.0/24 \
  --priority 200

Troubleshooting

Routes Not Taking Effect

If traffic is not being routed as expected:

  1. Trigger a config sync: arctic cluster sync
  2. Verify NFTables rules: nft list ruleset | grep arctic

Invalid CIDR Notation

Ensure CIDRs are valid:

  • Use slash notation: 10.0.0.0/8 not 10.0.0.0
  • Network address must match the mask: 10.0.0.0/8 not 10.1.2.3/8

Route Not Found

If a route ID is not found:

  1. List routes to verify: arctic routes list --service SERVICE_ID
  2. Ensure you are using the correct service ID

Set bandwidth limits

This section shows you how to configure bandwidth limits on Arctic services for Quality of Service (QoS) control.

Before you start

Ensure you have:

  • An existing service or the information to create one
  • Knowledge of the desired bandwidth limit in Mbps

Set bandwidth during service creation

Specify a bandwidth limit when creating a new service:

arctic services create \
  --target-peer TARGET_PEER_ID \
  --bandwidth-limit 1000
curl -X POST http://AGENT_IP:8080/v1/services \
  -H "Authorization: Bearer $TOKEN" \
  -H "Content-Type: application/json" \
  -d '{
    "target_peer_id": "TARGET_PEER_ID",
    "bandwidth_limit_mbps": 1000
  }'

The value is in Megabits per second (Mbps). 1000 means 1 Gbps.

Update bandwidth on existing service

Change the bandwidth limit on an existing service:

arctic services update SERVICE_ID --bandwidth-limit 500
curl -X PUT http://AGENT_IP:8080/v1/services/SERVICE_ID \
  -H "Authorization: Bearer $TOKEN" \
  -H "Content-Type: application/json" \
  -d '{"bandwidth_limit_mbps": 500}'

Remove bandwidth limit

Set the limit to 0 to remove the restriction (unlimited):

arctic services update SERVICE_ID --bandwidth-limit 0
curl -X PUT http://AGENT_IP:8080/v1/services/SERVICE_ID \
  -H "Authorization: Bearer $TOKEN" \
  -H "Content-Type: application/json" \
  -d '{"bandwidth_limit_mbps": 0}'

Verify current bandwidth

Check the current bandwidth setting:

arctic services get SERVICE_ID
curl -X GET http://AGENT_IP:8080/v1/services/SERVICE_ID \
  -H "Authorization: Bearer $TOKEN"

Look for the bandwidth_limit_mbps field in the output.

Common bandwidth values

Use CaseBandwidthValue
Low priority100 Mbps100
Standard1 Gbps1000
High throughput10 Gbps10000
UnlimitedNo limit0

How bandwidth limiting works

Arctic uses traffic shaping through the TProxy service to enforce bandwidth limits:

  1. Traffic Classification: Packets are marked with fwmarks for QoS
  2. Token Bucket: Traffic shaping uses a token bucket algorithm
  3. Fair Queuing: Multiple flows share the bandwidth fairly

Bandwidth limits apply to traffic flowing through the service. They do not affect traffic that does not match the service's routes.

Troubleshooting

Bandwidth Not Being Enforced

If traffic exceeds the configured limit:

  1. Trigger a config sync: arctic cluster sync
  2. Check agent logs: journalctl -u arctic | grep tproxymgr

Performance Lower Than Expected

If throughput is below the configured limit:

  1. Check for network bottlenecks elsewhere in the path
  2. Verify the underlying network can support the desired bandwidth
  3. Consider TCP tuning for high-bandwidth scenarios

Delete a service

This section shows you how to delete an Arctic service and clean up its associated configuration.

Before you start

Understand that deleting a service:

  • Removes all routes associated with the service
  • Removes the MACVLAN interface (if one was created)
  • Updates firewall rules to stop routing traffic
  • Does not affect the source or target peers

Steps

1. Find the service ID

List services to find the ID:

arctic services list
curl -X GET http://AGENT_IP:8080/v1/services \
  -H "Authorization: Bearer $TOKEN"

2. Delete the service

arctic services delete SERVICE_ID

# You will be prompted to confirm. Use --yes to skip confirmation:
arctic services delete SERVICE_ID --yes
curl -X DELETE http://AGENT_IP:8080/v1/services/SERVICE_ID \
  -H "Authorization: Bearer $TOKEN"

3. Verify deletion

Confirm the service was removed:

arctic services list

The service should no longer appear in the list.

What happens when you delete

When a service is deleted:

  1. Routes Removed: All CIDR-based routing rules are deleted
  2. Interface Cleaned Up: The MACVLAN interface is removed from the host
  3. Firewall Updated: NFTables rules no longer match traffic for this service
  4. Configs Regenerated: TProxy and IP tunnel configs are updated

Configuration changes propagate automatically, typically within seconds.

Bulk deletion

To delete multiple services, you can use a loop:

# Delete all services targeting a specific peer
for id in $(arctic services list --target-peer PEER_ID -j | jq -r '.[].id'); do
  arctic services delete "$id" --yes
done

Troubleshooting

Service Not Found

If the service ID is not found:

  1. Verify the service exists: arctic services list
  2. Check you are connected to the correct agent
  3. The service may have already been deleted

Interface Not Removed

If the MACVLAN interface persists after deletion:

  1. Check Network Manager logs: journalctl -u arctic | grep netmgr
  2. Manually remove if needed: ip link delete INTERFACE_NAME
  3. Trigger a sync: arctic cluster sync

Routes Still Active

If traffic is still being routed after deletion:

  1. Trigger a config sync: arctic cluster sync
  2. Verify NFTables rules: nft list ruleset | grep arctic
  3. Check agent logs for config updates: journalctl -u arctic | grep tproxymgr