Runner Provisioners: Autoscale Runners on Demand
Understand how the Docker provisioner creates, enrolls, and assigns single-job runner instances to meet changing job demand.
A provisioner watches how many jobs are waiting. It creates a durable runner instance before it starts ephemeral compute, then the runner enrolls and is assigned to a reserved job. Each activated runner claims at most one job and exits.
Use a provisioner when demand changes enough that a fixed runner fleet would sit idle or run out of capacity. One provisioner scales runner containers to the work.
Shipfox currently ships a Docker provisioner. Kubernetes and cloud virtual machine backends are on the roadmap.
How it works
The provisioner runs a continuous control loop against the Shipfox API:
Advertise capability
The provisioner reports labels and free slots for each template. Free slots equal
max_concurrency minus the containers already starting or running. A matching
workspace-owned provisioner always takes precedence over installation fallback,
even when it is busy or reports zero free slots.
Poll for demand
It long-polls the API. The API grants reservations when waiting jobs need labels that a template provides.
Create runner instances
The provisioner creates one runner instance and one single-use bootstrap token
(sf_rbt_…) for every planned runner. The bootstrap token grants only enrollment:
it does not reveal a workspace, reservation, or job credential.
Launch, enroll, assign, and reconcile
It starts one container per runner instance. The container exchanges its bootstrap token for a workspace-neutral control session, declares its labels, and waits. The provisioner assigns an enrolled instance only to its own reservation. The API then issues an activation token for that immutable assignment.
The provisioner reports each container's state. After a restart, it compares its
records with Docker and reconciles failed, expired, and unused runners. A template
can also keep target_concurrency enrolled runners ready without a reservation.
If Docker is unreachable, it advertises zero capacity and backs off.
Each provisioned runner claims one job, runs it, and exits. A burst starts more runners. Quiet periods leave no runner containers idle unless the template deliberately keeps a small prewarmed pool.
Templates
A template maps a label set to the container that provides it. Templates live in a YAML file the provisioner loads at startup:
templates:
docker-ubuntu22-2vcpu:
labels: [ubuntu22, ubuntu22-2vcpu] # labels the started runner registers with
image: shipfox-runner:ubuntu22 # container image to start
cpu: 2
memory: 4GiB
max_concurrency: 100
target_concurrency: 0When several templates satisfy a job's labels, the provisioner chooses the one with the fewest virtual CPUs. A specificity rule breaks a tie. The full template schema and every environment variable are in the Runner Provisioner reference.
Related pages
Run a provisioner
Token creation, templates file, and the launch command.
Runner Provisioner reference
Environment variables and the template schema.
Runners
Manually started runners, labels, and registration.