Concepts

Routing

How a job finds the right machine, and why tools run where the job runs.

Routing uses Temporal task queues. Nodes poll only the queues they qualify for, and Temporal hands each task to the first free poller.

The queues

QueuePolled byCarries
agent.<name>Every node of the tenant that runs the agent's plugin and has the labels the agent requiresThat agent's jobs
node.<id>One nodeTool calls for jobs whose home is that node, and other work that must run there
oz0.defaultEvery nodeWork for any node, such as tool calls for tools with affinity: any
Extra queuesNodes whose queues label lists themWork an operator wants on a chosen set of nodes

An agent's name is unique within a tenant, so agent.<name> always means one definition.

Labels decide where agents run

An agent can require labels:

agents/trainer.yaml
requires:
  labels: {gpu: "a100"}

Only nodes with gpu=a100 poll agent.trainer. Labels come from the join token or from the operator who accepts the node. Plugins can be limited to nodes with a label as well, with plugin install --select gpu=a100.

Tools follow the job

The node that picks up a job is its home node. Tools with the default affinity, node, run on the home node's queue, so an agent can work with files, devices and local models on that machine. The home is chosen when the job starts and recorded in its history, so it stays the same if the job's steps move between nodes.

If the home node goes away, the job's next tool call waits for it. Re-binding the job to another node that has the same agent, after a timeout, is designed (ADR 0008) and comes later.

Tools with affinity: any run on any node that has the plugin. Use it for stateless tools such as a web fetcher.

Routing pools such as agent.<name>~<pool>, which group nodes by a label selector without one queue per combination, are designed for larger fleets and come later.
Copyright © 2026