Feedback loops

A feedback loop runs part of a job again when a check fails. For example, an agent step changes the code. A later step runs the tests. If the tests fail, the job goes back to the agent step.

How a loop runsCopy Anchor Link

This complete workflow shows a loop. The project needs an npm test command and default agent settings.

# yaml-language-server: $schema=https://www.shipfox.io/docs/workflow.schema.json
name: Fix failing tests
runner: shipfox

triggers:
  manual:
    source: manual

jobs:
  fix:
    steps:
      - key: fix
        prompt: Fix the failing tests in this repository.

      - key: test
        run: npm test
        gate:
          success: step.exit_code == 0
          on_failure:
            restart_from: fix

This is what happens:

1. The fix step runs. The agent changes the code.
2. The test step runs npm test.
3. If the tests fail, the job goes back to the fix step. The files that the agent changed are still there, so the agent can continue from them.
4. The fix and test steps run again.
5. When the tests pass, the job continues.

The Gate fields reference lists the exact fields.

Check and restart are separateCopy Anchor Link

A gate has two parts. You write each part separately:

• success is the check. It decides if the step passed.
• on_failure is the restart rule. It names the step that runs again when the check fails.

Because the two parts are separate, a failed check has two possible results:

• Stop the job. Write only success. Do this when it is not safe to run the steps again.
• Go back to the step that can fix the problem. Add on_failure and name that step. The job does not start again from its first step.

Restart or failCopy Anchor Link

Some failures are worth another try. Some are not. An earlier step can fix a merge conflict, so the job should restart. No step can fix a missing credential, so the job should fail.

To choose, give on_failure a list of rules. Each rule has an if condition. Shipfox checks the rules in order and uses the first rule where if is true.

steps:
  - key: select_branch
    run: ./scripts/select-branch.sh

  - key: prepare
    run: ./scripts/prepare.sh

  - key: publish
    run: ./scripts/publish.sh
    outputs:
      failure_kind:
        type: string
        from_file: failure-kind.txt
    gate:
      on_failure:
        - if: has(step.outputs.failure_kind) && step.outputs.failure_kind == "foreign-commits"
          restart_from: select_branch

        - if: has(step.outputs.failure_kind) && step.outputs.failure_kind in ["stale-head", "transient"]
          restart_from: prepare
          max_attempts: 2

The publish script writes the kind of failure to failure-kind.txt. When publish fails:

• For foreign-commits, the job goes back to select_branch.
• For stale-head or transient, the job goes back to prepare.
• For any other kind, no rule applies. The job fails, and publish keeps its own error.

When no rule applies, the job fails. A failure that you did not plan for does not restart by accident.

Keep these points in mind:

• if can read step.exit_code, step.status, and step.outputs.
• A step can fail before it sets an output. Use has() before you read an output. If Shipfox cannot read the value, the job fails.
• A rule with no if always applies. Put it last.
• A gate with success set to true passes the step, and Shipfox reads no rule. To choose only with rules, leave success out. The status of the step then decides if the step passed.

Use a separate checkCopy Anchor Link

Do not ask the agent that did the work to also check the work. An agent that made a mistake often does not see that mistake when it checks.

Use a check that gives the same answer for the same code, whatever the agent says. Good checks are:

• A test command.
• A type check.
• A linter.
• A review by a different agent step.

Route several loops by their checking stepCopy Anchor Link

One job can contain more than one feedback loop. In this example, the job has two checks, test and lint. Each check has its own fix step:

steps:
  - key: fix_tests
    if: '${{ has(step.restart) && has(step.restart.from.key) && step.restart.from.key == "test" }}'
    prompt: Fix the failing tests.

  - key: test
    run: npm test
    gate:
      success: step.exit_code == 0
      on_failure:
        restart_from: fix_tests

  - key: fix_lint
    if: '${{ has(step.restart) && has(step.restart.from.key) && step.restart.from.key == "lint" }}'
    prompt: Fix the lint errors.

  - key: lint
    run: npm run lint
    gate:
      success: step.exit_code == 0
      on_failure:
        restart_from: fix_lint

step.restart.from.key is the key of the check that failed. Each fix step tests this value in its if condition. The two has() tests make sure that the value exists before the condition reads it:

• When test fails, the job goes back to fix_tests. fix_tests runs.
• When lint fails, the job goes back to fix_lint. fix_lint runs.
• On the first attempt, no check has failed. Both fix steps are skipped.

Build a feedback loop shows the complete workflow.

Tell the next attempt what failedCopy Anchor Link

When a gate restarts an agent step, Shipfox tells the agent what failed.

To give the agent more details, add feedback to the gate:

steps:
  - key: fix
    prompt: Fix the failing tests.

  - key: test
    run: npm test
    gate:
      success: step.exit_code == 0
      on_failure:
        restart_from: fix
        feedback: Fix the code, not the tests.

Loop boundsCopy Anchor Link

Two limits stop a loop that does not pass its check:

# yaml-language-server: $schema=https://www.shipfox.io/docs/workflow.schema.json
name: Fix failing tests
runner: shipfox

triggers:
  manual:
    source: manual

jobs:
  fix:
    execution_timeout: "30m"
    steps:
      - key: fix
        prompt: Fix the failing tests.

      - key: test
        run: npm test
        gate:
          success: step.exit_code == 0
          on_failure:
            restart_from: fix
            max_attempts: 8

• max_attempts is the number of times the test step can run. The first run counts. Here, if test fails 8 times, the job fails with restart_exhausted. The default is 5. You cannot remove the limit.
• execution_timeout is the time that the job can run. The time includes every attempt. Here, if the job is not complete after 30 minutes, it fails with timed_out.

With a list of rules, each rule has its own max_attempts. The number counts every run of the step. It does not count only the runs that this rule started.

steps:
  - key: fix
    prompt: Fix the problem.

  - key: test
    run: ./scripts/test.sh
    gate:
      on_failure:
        - if: step.exit_code == 2
          restart_from: fix

        - if: step.exit_code == 3
          restart_from: fix
          max_attempts: 2

This is what happens:

1. test exits with code 2. The first rule applies, and the job restarts from fix.
2. test runs a second time and exits with code 3.
3. The second rule applies. It has max_attempts: 2, and test has already run 2 times.
4. The job does not restart. It fails with restart_exhausted.

The limit that the job reaches first stops the loop.

Limits lists the exact values.

Avoid repeated writesCopy Anchor Link

When a gate fails, the job runs again every step from the restart_from step to the step with the gate. A step in that range that pushes, deploys, or comments does it again on each attempt.

Put such a step after the gate. It then runs one time, after the check passes:

steps:
  - key: fix
    prompt: Fix the failing tests.

  - key: test
    run: npm test
    gate:
      success: step.exit_code == 0
      on_failure:
        restart_from: fix

  - key: push
    run: git push origin HEAD

Here, a restart runs fix and test again. push is after the gate, so it runs only when test passes.

A tool step can send the job back to an earlier step only when it reads data. If the gate of a tool step that writes data fails, the job fails. For an example, see Wait for a change to apply.

If a step that writes must stay in the loop, make it safe to run again. For example, a push step must use the branch that an earlier attempt created, and not fail because the branch exists. Turn a Sentry issue into a verified pull request shows a push step that does this.

To add a feedback loop to a workflow, see Build an agent feedback loop.