Install and update

Install

orchestrator-zero plugin install https://github.com/acme/oz-github-tools

Without --ref, the edge takes the highest vX.Y.Z tag, compared as numbers, or the default branch when there is no release tag. Pin a branch, a tag or a full commit SHA with --ref:

orchestrator-zero plugin install https://github.com/acme/oz-github-tools --ref v1.4.0
orchestrator-zero plugin install https://github.com/acme/oz-github-tools --ref main --select team=platform

--select key=value installs it only on nodes with that label; repeat it to require several. --tenant picks the tenant (default default). Installing a plugin that is already installed updates it.

In dev mode the edge also accepts file:// URLs and local folders.

A plugin in a folder

A repository can hold several plugins, each in its own folder with its own oz0-plugin.yaml. Name the folder with --path:

orchestrator-zero plugin install https://github.com/acme/agents --path plugins/github-tools

The ref still applies to the whole repository: the edge locks the commit, then packs only that folder, so the plugin sees the folder as its root and nothing else from the repository reaches the nodes. The path cannot leave the repository, and a folder that is a symlink does not count as one.

A private repository

The edge fetches a private repository over HTTPS with a token you keep as a tenant secret. Store the token, then name the secret with --secret:

orchestrator-zero secret set GITHUB_TOKEN --from-file ~/.secrets/github-token
orchestrator-zero plugin install https://github.com/acme/private-tools --secret GITHUB_TOKEN

The secret holds the token alone, or user:token for Git servers that need a particular user name. A token alone goes as the user x-access-token, which GitHub accepts for personal access tokens and app tokens alike. Give it read access to that repository only: on GitHub, a fine-grained token with Contents: read.

The edge unseals the token for the fetch and hands it to git in its environment, never on a command line, and only for the repository's host, over HTTPS. The plugin remembers the secret's name, never its value, so plugin update fetches the same way; after you rotate the token with secret set, updates use the new one.

A quality gate for plugins with evals

A version of a plugin that has evals runs them on one test node before the fleet gets it, and is stopped when it does worse than its thresholds or the version it replaces. Until then the fleet keeps the installed version:

Testing team 0.2.0 (commit ff4d7eae2c6d…) in tenant default before the fleet gets it: its evals run on a test node.
Until it passes, the fleet keeps team 0.1.0.

--no-gate installs it at once. See Quality gates for how the verdict works.

See how it is going

orchestrator-zero plugin list

plugin list shows every plugin, its locked commit (and its folder, for one in a folder), and each node's state with it: whether the node has built and started it, the functions it found, or the error it hit. For a plugin with evals it also shows the gate of its latest version: waiting, testing, passed or stopped, and why.

Update

orchestrator-zero plugin update github-tools

update resolves the plugin's ref again, in the same folder and with the same secret, and rolls out what it points to now, through its quality gate when it has evals. With --ref main, that is the latest commit on main; with a tag, the same commit unless the tag moved. To move to a new release, install it again with the new --ref.

A node restarts its runtime once the new version is ready. If the new version fails to build or start, the node keeps the previous one running and reports the failure in plugin list.

New versions roll out to canaries first

When a plugin runs on two or more online nodes, its new version does not reach them all at once. It goes to a canary node first, then to the rest in steps, and each step must pass the same health and quality gates as a runtime rollout. A version with evals passes its quality gate first, then rolls out.

orchestrator-zero plugin update github-tools
orchestrator-zero plugin install https://github.com/acme/oz-github-tools --ref v1.5.0 --canary 2 --batch 5 --soak 10m

Rolling team 0.2.0 (4f1c2d9) out in tenant default, canaries first: rollout rollout-6b1e0c2a.
Until it is done, the other nodes keep team 0.1.0.
Follow with: orchestrator-zero rollout show rollout-6b1e0c2a (--all-at-once gives a version to every node at once)

Each step:

1. gives the new version to its nodes, which run it instead of the installed one, and waits until they report it ready, for at most --start-timeout (5 minutes);
2. sends those nodes their share of the plugin's new jobs: with one node of three on the new version, a third of them (the nodes of a step that is starting take no new jobs until they run the new version);
3. watches them for --soak (1 minute), and then judges the step.

The new version answers on task queues of its own, roll.<commit>.agent.<name> and roll.<commit>.flow.<name>, so each job runs on one version from its start to its end, and the quality gate can tell them apart. It compares the jobs that failed on those queues with the jobs that failed on the plugin's usual queues over the same time. A child job from the same plugin stays on its parent's version. A node takes the new version's jobs once it runs the new version, and in the last step, when no node is left on the old version, the rollout's nodes also finish the jobs left on the usual queues.

When the last step passes, the new version becomes the installed version on every node. When a step fails, or you abort with rollout abort, the candidate is dropped and the canaries go back to the installed version. Either way, the nodes on the installed version finish the jobs that are still on the rollout's queues; rollout show says draining until none is left. Here a version that changed its agent's input contract failed its canary's jobs:

Rollout rollout-838939ab: tenant default, plugin team from 0.2.0 (25e707a) to 0.3.0 (dc6f7d2), rolled-back, draining
  canary  failed   node-030651b413c5: step 1: 54 of 54 jobs failed on 0.3.0 (100%), against 0% on 0.2.0
  step 2  pending  node-d81206129a00
  step 3  pending  node-e4c46863afd8
Reason: step 1: 54 of 54 jobs failed on 0.3.0 (100%), against 0% on 0.2.0

In make e2e-rollout, which runs jobs in four loops meanwhile, the rollout ended 28 seconds after plugin install, and only the jobs that reached the canary failed: 60 of 233.

• A tenant runs one rollout at a time, of a runtime or of a plugin. A plugin rollout asked for while another one runs waits for its turn, and its steps are planned when it starts.
• While a plugin has a rollout waiting or running, a new version of it is refused; wait for the rollout to end, or abort it.
• A first install, a plugin on one node, and --all-at-once go to every node at once.
• An agent the new version adds can be started once the rollout is done. Jobs check their input against the installed version's schema.
• A job whose plugin changed under it in a way its history cannot follow, such as a flow that lost a step, fails with the reason instead of waiting forever.

The Plugins page shows each rollout's steps next to the plugin, with Roll back, or Drop for one that waits. See ADR 0052 for the design.

Remove

orchestrator-zero plugin remove github-tools

Nodes stop the plugin's processes and delete it. A rollout of it that waits or runs ends.