Runner Provisioners: Autoscale Runners on Demand
Understand how the Docker provisioner creates, enrolls, and assigns single-job runner instances to meet changing job demand.
A provisioner watches how many jobs are waiting. It creates a durable runner instance before it starts ephemeral compute, then the runner enrolls and is assigned to a reserved job. Each activated runner claims at most one job and exits.
Use a provisioner when demand changes enough that a fixed runner fleet would sit idle or run out of capacity. One provisioner scales runner containers to the work.
Shipfox ships a Docker provisioner.
How it works
The provisioner runs a continuous control loop against the Shipfox API:
Advertise capability
The provisioner reports labels and free slots for each template. Free slots equal
max_concurrency minus the containers already starting or running. A matching
workspace-owned provisioner always takes precedence over installation fallback,
even when it is busy or reports zero free slots.
Poll for demand
It long-polls the API. The API grants reservations when waiting jobs need labels that a template provides.
Create runner instances
The provisioner creates one runner instance and one single-use bootstrap token
(sf_rbt_…) for every planned runner. The bootstrap token grants only enrollment:
it does not reveal a workspace, reservation, or job credential.
Launch, enroll, assign, and reconcile
It starts one container per runner instance. The container exchanges its bootstrap token for a workspace-neutral control session, declares its labels, and waits. The provisioner assigns an enrolled instance only to its own reservation. The API then issues an activation token for that immutable assignment.
The provisioner reports each container's state. After a restart, it compares its
records with Docker and reconciles failed, expired, and unused runners. A template
can also keep target_concurrency enrolled runners ready without a reservation.
If Docker is unreachable, it advertises zero capacity and backs off.
Each provisioned runner claims one job, runs it, and exits. A burst starts more runners. Quiet periods leave no runner containers idle unless the template deliberately keeps a small prewarmed pool.
Templates
A template maps a label set to the container that provides it. Templates live in
a YAML file the provisioner loads at startup. A file can share lookup data through
vars, apply a fragment through defaults, keep genuine one-offs under templates:,
and generate variants through independent matrix families.
defaults:
target_concurrency: 0
templates:
docker-local-debug:
labels: [docker, local-debug]
cpu: 1
memory: 2GiB
max_concurrency: 2
cost: 1
matrix:
general:
axes:
os: [ubuntu22, ubuntu24]
cpu: [2, 4]
template:
labels: [docker, "${{ os }}", "${{ cpu }}vcpu"]
image: "ghcr.io/shipfoxhq/runner:${{ os }}"
cpu: "${{ cpu }}"
memory: "${{ cpu * 2.0 }}GiB"
max_concurrency: 50
cost: "${{ cpu }}"
gpu:
axes:
cuda: [cuda12, cuda13]
memory: [16g, 32g]
template:
labels: [docker, gpu, "${{ cuda }}", "${{ memory }}"]
image: "ghcr.io/shipfoxhq/runner:${{ cuda }}"
cpu: 8
memory: "${{ memory }}"
max_concurrency: 10
cost: 20The two matrix blocks are independent families. Add another hardware class by adding
another block with its own axes; do not add unrelated axes to the general fleet.
Labels may overlap across families. Matching is subset-based, and the lowest cost
wins when several templates can serve a job. A hand-written entry whose key matches a
generated key shadows that generated variant and is the per-variant override idiom.
defaults is a deep merge for maps, but lists and scalars replace wholesale. If a
family needs a different subnet list or security-group list, set the full list in that
family. The full template schema and every environment variable are in the
Runner Provisioner reference.
Docker template files use strict schemas. Unknown top-level or template keys that may have been silently ignored before now fail with a file-scoped error, so remove them when upgrading an existing file.
Related pages
Run a provisioner
Token creation, templates file, and the launch command.
Runner Provisioner reference
Environment variables and the template schema.
Runners
Manually started runners, labels, and registration.