Troubleshooting
Start with --log-level debug on the server or the node, and --json on CLI commands for the full picture.
Nodes
join fails with a certificate or hash error
The node checks the server against the CA hash in the token or in --ca-hash before it sends anything. Check that:
- the hash matches
orchestrator-zero cluster info; --serveris an address on the edge's certificate: the server's--advertiseaddress, or a name you gave it with--san;- the token has not expired or been revoked (
orchestrator-zero token list).
A node waits for approval forever
It joined without a token. Accept it: orchestrator-zero node list --pending, then orchestrator-zero node accept <node-id>.
A node stopped working and says it is blocked
An operator blocked it. Check with orchestrator-zero node show <node-id> and let it back in with node unblock; the node notices within ten seconds.
A node cannot reach the edge
Nodes need outbound access to the edge on 7233 and 7443, and nothing else. Check firewalls and proxies between them. A node with several edges in its list tries the others on its own.
Agents and jobs
A job stays RUNNING and nothing happens
No node is ready to run the agent. orchestrator-zero catalog shows the nodes ready for each agent. Check that the plugin is installed for the node (plugin list, and --select if you used it), and that the node has every label in the agent's requires.labels. The job starts as soon as a node can take it.
A job waits after a node went away
Its tools run on its home node, the node that picked it up. If that node is gone, the job's next tool call waits for it to come back. Bring the node back, or cancel the job and start it again.
A job fails with a model error
When a model call fails, the job's error starts with what kind of failure it was, followed by the provider's own reason:
| Error | What happened | What to do |
|---|---|---|
MissingKey | The tenant has no key for any of the agent's models, so the gateway refused every call with 412 | Set the key, as the error says, and run the job again |
ModelUnavailable | The provider was overloaded, rate limited or down, or the connection broke, on all five attempts | Run the job again later, or give the agent a fallback model on another provider |
ModelRejected | The provider refused the request itself: a key it does not accept (401), no access (403), a model name it does not know (404), or a request that is invalid (400) or too large (413) | Fix what the provider's reason names; another attempt would fail the same way |
SpendLimitReached | The provider account's spend cap or quota is used up | Raise the limit with the provider, or wait until it resets |
BudgetExceeded | The job tree spent its budget | See Contracts and limits |
For a missing key:
printf %s "$ANTHROPIC_API_KEY" | orchestrator-zero secret set ANTHROPIC_API_KEY
Only ModelUnavailable is retried, by Temporal alone: five attempts, each one call to each of the agent's models, waiting as long as the provider's retry-after header asks. The others fail the job after one call.
The job failed with InvalidInput
The input does not match the agent's input_schema. The error says where. With an input schema, pass JSON: orchestrator-zero run reviewer '{"repo": "acme/api", "pr": 42}'.
The job ended at a limit
It reached max_steps, max_tokens or timeout. Raise the limit in the agent definition if the work really needs more, or make the instructions more focused.
Plugins
plugin list shows an error on a node
The node could not build or start the plugin, and kept the previous version running. The error says why. Common causes: a dependency that does not build on that platform (check requires.platforms), a uv.lock that does not match pyproject.toml (run uv lock and commit it), or an MCP command that does not start (try it by hand with uv run).
plugin install says the repository cannot be reached
Only public repositories install today, over HTTPS. Check the URL and that the edge can reach the Git host.
orchestrator-zero dev cannot find the node or the runtime bundle
It needs orchestrator-zero-node next to the server binary or on PATH (or --node-binary), and a runtime bundle: --runtime-bundle, $OZ0_RUNTIME_BUNDLE, or the newest in ./dist/runtime. Build one with make runtime-bundle.
Servers
The server refuses a port
Temporal's ports must be 32767 or lower, because its PostgreSQL schema stores them in a small integer column. The defaults are fine.
The server refuses to start on the database
The database's schema is newer than this binary knows: another server was upgraded first. Upgrade this one too.
Temporal reports too many connections
Give PostgreSQL a max_connections of at least 100 per server.