Feedback loops and retry limits
Why a run goes back; the revise loop, check repair hops, the intake repair, the hop cap, attempt retries and tool retries, each with a default, where it is set and what happens at the limit.
A run goes back for six reasons, each with its own counter. This page gives every number, where it is set and what happens when it runs out. File:line citations are in parentheses; "not checked" means the code was not read for that fact. Numbers marked with a fact marker are pinned by tests, so a changed default fails a test until this page changes.
01review send-backs and the revise loop#
When review does not accept, the baton goes to the revise_route router. It routes on needs_human: true goes to the gate, false goes back to work. max_false with exhausted_outlet: human limits the false route (internal/harness/ticket-completion.yaml:371-377, cords 490-493).
- The count is the number of times
revise_routetook theworkoutlet since the operator's last gate decision. A gate decision, approve or send back, resets it to zero. - A preset sets the limit. The launch replaces
max_falseonrevise_routewith the preset'smax_revise_loops. The shipped file says 2 (ticket-completion.yaml:374); whether a launch without a preset keeps that value is not checked. - The shipped presets:
| preset | max_revise_loops |
|---|---|
| quick | 1 |
| balanced | 2 |
| thorough | 2 |
| max | 3 |
| lean | 2 |
| heavy | 2 |
- When the loop runs out, the router sends the baton to the gate and adds a
loop_exhaustedline to it: "<node> took <outlet> <n> times; stopped looping" (nodes). The gate shows why it reopened. - A reviewer
needs_humanverdict does not count against the limit. It goes straight to the gate (ticket-completion.yaml:374,true_outlet: human). - Operator preset tweaks can change
max_revise_loops; a custom value is applied the same way.
02check repair hops#
When the checks fail, check_router sends the baton back to work for a repair, up to repair_bound times, then to its failed_outlet.
repair_bounddefaults to2hops. The shippedcheck_routersets no value, so the default applies (ticket-completion.yaml:221-222, registry text at).- Each hop is journaled as
repair.hopwith the hop number and the bound. - When the hops are used up, the node reports
checks.terminal_failurewith "checks failed after N of M repair hops" and emits onfailed_outlet. - An environment failure never repairs. If every failing check failed because its command is missing or not executable (exit 126 or 127, or "command not found"), the node ends instead of routing to repair. A
stage_errorverdict also goes tofailed_outletas environmental. See checks.
03intake capability-assert repair#
The intake_proof node (capability_assert) checks that intake made the calls it must. If the only call missing is its repair_tool (rate_search_result in the shipped template), it sends the source node back once, with repair_field set to an instruction (ticket-completion.yaml:53-76).
- The "once" is logic, not a constant: the node looks for an earlier
node.observedevent from itself with the reason "capability repair hop" and does not repeat. So there is no number to pin. - Any other miss fails the node with the assertion's own error.
- This repair is a node observation. It is not a
repair.hopevent, so it does not count againstrepair_bound.
04the hop cap#
A loop that never converges is ended by the run's hop cap. See the hop cap for the journal side.
- A launched run's cap is
32hops. The scheduler's own default is 64 and only applies outside harness launches. HopResetBylists everyhuman_gatenode of the template. Each such node's decision counts hops from zero again.HopLimitTois set only when the template has exactly one human gate. An envelope that would cross the cap then goes to that gate andfeedback.limit_divertedis journaled. With no gate or several,feedback.limit_reachedis journaled and the run stops (event names at).
05attempt retries#
A model node runs as one or more attempts under an AttemptPolicy. Retrying is opt-in. An omitted policy resolves to one attempt and no hidden recovery.
| field | default | source |
|---|---|---|
restart |
never |
|
max_attempts |
1 |
|
backoff.initial |
1s |
, 104-106 |
backoff.maximum |
30s |
, 107-109 |
backoff.multiplier |
2 |
, 110-112 |
rate_limit_wait |
6h |
restart: on_failureneedsmax_attemptsof at least 2, an explicitretryable_reasonslist and anactive_deadline.restart: neverrequiresmax_attempts1 and no reasons.- Only these terminal reasons can be listed as retryable: deadline exceeded, task limit, memory limit, CPU limit, output limit, launch failure, protocol failure and host loss.
- A retry is journaled as
attempt.retry_scheduled, with the backoff before the next attempt (runs). - A rate-limited attempt is parked until the limit resets, for at most
rate_limit_wait, without holding a scheduler slot (runs).
06tool retries#
A declared tool (harness.json tools[].retries) retries 0 times when unset. A negative value is refused at load.
07where to read them#
run_query sendbacks lists each send-back of a run, a rejected gate or a node's revise outlet, with its round and why (; it keeps the last 8 at). run_summary counts retries, repair hops and send-backs (runs). Gate behaviour is on gates.
08when you're the conductor#
- A gate that carries a
loop_exhaustedline means the automatic loop is spent. Another send-back starts a fresh count, so say what is different before proposing one. - Do not read a
needs_humangate as a spent loop. It reached the gate on its first review. - Read
run_query sendbacksbefore explaining why a run went back. Name the number from this page that applies, not a guess. - Never promise more repair hops than
repair_bound, or more revise passes than the run's preset allows.