Runner Provisioner Configuration Reference

This page is the canonical reference for configuring the Docker runner provisioner (@shipfox/provisioner-docker). For how a provisioner works, see Runner Provisioners; for starting one, see Run a provisioner.

Required variables

Optional variables

Docker logging and failed-container forensics

Provisioner logs and runner-container output are separate streams. The provisioner never copies runner stdout or stderr into its own logs. Use LOG_LEVEL, LOG_PRETTY, LOG_STDOUT, LOG_STDOUT_LEVEL, LOG_FILE, and LOG_FILE_LEVEL for provisioner-process logs. LOG_FILE writes a plain file without built-in rotation, so configure external rotation such as logrotate. The logger keeps the file open: use copytruncate in the rotation rule or restart the provisioner after rename/create rotation, otherwise it continues writing to the old inode. For production, prefer stdout with journald or runtime-managed rotation.

Runner containers inherit the Docker daemon logging driver unless SHIPFOX_PROVISIONER_DOCKER_LOG_DRIVER is set. A local driver with on-host rotation can be configured as follows:

export SHIPFOX_PROVISIONER_DOCKER_LOG_DRIVER=local
export SHIPFOX_PROVISIONER_DOCKER_LOG_OPTIONS='{"max-size":"10m","max-file":"5"}'

For journald, configure the driver and a tag, then retrieve output with journalctl:

export SHIPFOX_PROVISIONER_DOCKER_LOG_DRIVER=journald
export SHIPFOX_PROVISIONER_DOCKER_LOG_OPTIONS='{"tag":"shipfox-runner/{{.Name}}"}'
journalctl CONTAINER_NAME=<runner-name> --since today

Enable persistent journald storage (Storage=persistent in /etc/systemd/journald.conf) and configure its disk and rotation limits. Remote drivers use the retention and access policy of their durable backend. For local and json-file, docker logs --timestamps --tail 200 <runner-name> works only until cleanup removes the container. Use journald or the remote backend after cleanup.

Failed containers remain for forensic inspection only when both retention settings are greater than zero. The default TTL is one hour and the default count bound is 20. Docker FinishedAt drives TTL cleanup. Missing FinishedAt uses the first observed failure time instead of creation time, so a long-running container is not removed immediately. Terminal inspection failures defer TTL cleanup but remain subject to the count bound, and unknown failure times rank as newly observed for count eviction. Successful exits, dead containers, stale-created containers, and backend termination keep their immediate cleanup behavior.

Forensic output is driver-specific: use docker inspect <runner-name> and docker logs --timestamps --tail 200 <runner-name> for local or json-file, journalctl CONTAINER_NAME=<runner-name> for journald, and the configured durable backend for remote drivers. For a remote Docker daemon, set DOCKER_HOST to the configured SHIPFOX_PROVISIONER_DOCKER_HOST value before running Docker commands; omit it for the local default. Run journalctl on the Docker daemon host, not on the operator workstation. The none driver has no container output.

Template file schema

The templates file can contain shared lookup data, a default fragment, hand-written one-offs, and any number of independent matrix families. This is a complete Docker example with a general fleet, a separate GPU family, and one hand-written template:

vars:
  image_by_os:
    ubuntu22: ghcr.io/shipfoxhq/runner:ubuntu22
    ubuntu24: ghcr.io/shipfoxhq/runner:ubuntu24

defaults:
  labels: [docker]
  target_concurrency: 0

templates:
  docker-local-debug:
    labels: [docker, local-debug]
    image: ghcr.io/shipfoxhq/runner:debug
    cpu: 1
    memory: 2GiB
    max_concurrency: 2
    cost: 1

matrix:
  general:
    axes:
      os: [ubuntu22, ubuntu24]
      cpu: [2, 4]
    template:
      labels: [docker, "${{ os }}", "${{ cpu }}vcpu"]
      image: "${{ vars.image_by_os[os] }}"
      cpu: "${{ cpu }}"
      memory: "${{ cpu * 2.0 }}GiB"
      max_concurrency: 50
      cost: "${{ cpu }}"

  gpu:
    axes:
      cuda: [cuda12, cuda13]
      memory: [16g, 32g]
    template:
      labels: [docker, gpu, "${{ cuda }}", "${{ memory }}"]
      image: "ghcr.io/shipfoxhq/runner:${{ cuda }}"
      cpu: 8
      memory: "${{ memory }}"
      max_concurrency: 10
      cost: 20

The checked-in Docker and EC2 files at apps/provisioner-docker/templates.example.yaml and apps/provisioner-ec2/templates.example.yaml use the same shape. Copy one, replace the placeholder images, AMI IDs, subnets, and security groups, then point SHIPFOX_PROVISIONER_TEMPLATES_FILE at the copy.

Matrix families

Families are independent. Adding a GPU pool means adding a gpu block with GPU axes; it does not mean adding GPU axes to the general fleet. Labels may overlap across families. Matching is subset-based, and the lowest cost wins when several templates can serve the same job, followed by specificity.

Defaults do not append lists. If defaults.subnets contains two general subnets and a GPU family sets subnets: [subnet-gpu], the GPU template has only subnet-gpu. Use a block override whenever a family needs a different list or scalar.

The provider validates every rendered template with a strict schema. For Docker, unknown top-level or template keys now fail with a file-scoped error. This is a breaking upgrade for existing Docker files that relied on mistyped keys being ignored; remove those keys before upgrading. The optional Docker cost field defaults to cpu.

Template selection: when several templates satisfy a job's labels, the provisioner picks the lowest cost (or cpu for Docker when cost is omitted), then the most specific. Use a hand-written entry under templates: to override one generated key; the hand-written value wins and the loader logs the shadowed variant.

Related pages