Run a cluster
A cluster is one or more orchestrator-zero servers sharing one PostgreSQL server. Each server runs every role, management, edge and Temporal, in one process. Servers find each other through the database; there is no seed list to maintain.
What you need
- PostgreSQL 13 to 16, reachable from every server. The user in the URL should be allowed to create databases; otherwise create the three databases yourself (see below).
max_connectionsof at least 100 per server. Temporal opens a connection pool per service and store.- The
orchestrator-zerobinary on every server, and a copy of the master key file once it exists. - Open ports between servers: 7234 to 7236 and 7239 (Temporal gRPC), 6933 to 6935 and 6939 (Temporal membership). Nodes need 7233 and 7443; operators need 8443.
Create the cluster once
export OZ0_DB='postgres://oz0:<password>@db.internal:5432/oz0?sslmode=require'
orchestrator-zero server init --cluster-name prod-eu --history-shards 1024 \
--master-key-out /etc/orchestrator-zero/master.key
server init creates three databases, named after the one in the URL (oz0, oz0_temporal and oz0_temporal_visibility), applies Temporal's schema migrations and ours, and creates:
- The master key file. Back it up and keep it secret. Every server needs a copy; without it the cluster's CAs and secrets cannot be unsealed and the cluster is lost.
- Two certificate authorities, sealed with the master key and stored in the database: the cluster CA (nodes, operators and the edge's ports) and the internode CA (only Temporal's traffic between servers).
- The first tenant,
default, with a Temporal namespace of the same name. - The first operator,
admin, whose certificate and CLI context go to~/.config/orchestrator-zero/<cluster>/. - The web UI's first account, also
admin, with a generated password thatserver initprints once. Sign in athttps://<server>:8443; see Web UI.
Choose the history shards carefully
The number of Temporal history shards can never change. 1024 suits a cluster that may grow to thousands of state transitions per second; small clusters can use 64 or 128.
Start the servers
On each server, with the same database URL and a copy of the master key:
export OZ0_DB='postgres://...'
orchestrator-zero server start --master-key-file /etc/orchestrator-zero/master.key --advertise 10.0.0.11
--advertise is the address other servers and nodes reach this server on. Add --san edge.example.com for every extra name nodes use to reach the edge, such as a load balancer.
Point the CLI at the cluster
export OZ0_CONTEXT=~/.config/orchestrator-zero/prod-eu/context.json
orchestrator-zero cluster info
cluster info shows the cluster's name, its CA hash and the servers that are up.
Operator certificates
An operator's certificate is valid for 90 days. Commands warn on stderr when it has less than 15 days left. Renew it while it works:
orchestrator-zero operator renew
The CLI makes a new key, the cluster signs a certificate for it, and both replace the ones in the context's directory. The audit log records it as operator.renew.
An operator whose certificate expired or was lost gets new credentials on a server's machine, from the database and the master key. The command also adds an operator the cluster does not have yet:
orchestrator-zero server operator issue admin --master-key-file /etc/orchestrator-zero/master.key \
--operator-dir ~/.config/orchestrator-zero/prod-eu --advertise oz0.example.com
A context already in --operator-dir keeps its addresses; a new one points at --advertise. Whoever can run it is trusted like a server, since it reads the master key. For a dev server, use --dev --data-dir <dir>.
Keys stay in memory
Each server issues itself certificates from the cluster's CAs at start and renews them every day, in memory. No server key is written to disk; holding the master key is what makes a machine a server.