ADR-0023: Completion continuations — resume a parked workflow when a watched process ends
Context and Problem Statement
ask-and-dispatch (ADR-0021) and script-driven immediate dispatch (ADR-0022) let a running command hand work to another agent, but their trigger is a human answering a question or a script calling an endpoint. A parked workflow — e.g. a LangGraph graph that spawned a Homer sub-run and now needs to resume when that sub-run finishes — has no Homer primitive to wake it on a sub-process's terminal state. We need a registration whose trigger is "a watched sub-process reached a terminal status," firing a target agent run that carries the watched process's result so the graph can resume.
Decision Drivers
- The parent must always wake — on success, failure, kill, or a restart that orphaned the run (
UNKNOWN). A missed completion silently strands the graph forever. - Exactly-once fire: a crash between run-start and the durable mark must not double-fire, and concurrent triggers (live hook + boot replay) must collapse to one run.
- The trigger must observe a durable, authoritative terminal status, never a best-effort signal.
- Reuse the dispatch machinery (opt-in gate, CAS state model, SQLite store, TTL sweep, boot reconciliation) rather than inventing a parallel one.
Considered Options
- Trigger off
ProcessEventBroadcastervs. trigger off the authoritative terminal-status commit inProcessExecutor. - Reuse dispatch's
CANCELLEDfor TTL expiry vs. a separateEXPIREDstate. - Always inline the watched result vs. cap inline size, fall back to a reference (
result = null). - Constructor-inject
ContinuationServiceintoProcessExecutorvs. lazy/provider injection to break the DI cycle.
Decision Outcome
Chosen:
Trigger off the authoritative terminal commit, not the broadcaster.
ProcessEventBroadcasterisDROP_OLDEST/lossy (a slow or absent consumer drops events), so a completion could be missed — fatal for a wake trigger. The hook runs fromProcessExecutor'sfinally, after the terminal status is durably committed (CAS for FINISHED, the exception path for FAILED, and KILLED preserved by the KILLED-wins invariant), re-reading the actual final status viagetProcessStatusexactly as telemetry does. Hook failures are caught and logged so they can never corrupt the process's terminal status.A separate
EXPIREDstate. Dispatch reusesCANCELLEDfor both operator cancel and TTL lapse; for continuations the two are worth distinguishing (an operator cancel vs. "the watched process never finished in time"), so TTL sweep moves PENDING →EXPIREDand explicit cancel moves PENDING →CANCELLED.Inline-result size cap with reference fallback. The watched process's final stdout is run through the optional
resultTransformerand embedded in the resume payload ({status, processId, result}). Over 64 KiB,resultisnull— the parent resumes by reference (it can fetch the artifact) rather than inflating the env/payload. (Superseded in part by ADR-0025: the over-cap branch now adds a fetchableresultRefcapability token instead of a bareprocessId.)Opt-in via
dispatch.enabledon the target agent, the same gate ask-and-dispatch uses — no new opt-in surface.Lazy/provider injection to break the DI cycle.
ContinuationService→AgentRunStarter→ProcessExecutor→ (terminal hook) →ContinuationServiceis a cycle Koin cannot satisfy by constructor injection.ProcessExecutortakes anonTerminal: suspend (Int, ProcessStatus) -> Unitcallback wired in DI to resolveget<ContinuationService>()at call time, so construction stays acyclic.Two distinct param channels on
POST /api/v1/callback/runs:params→ sub,continueParams→ continuation. Ahomer-dispatchcall carries two audiences of inputs — the dispatched sub-agent and the resumed parent (the continuation). The LangGraph runtime's per-interrupt resume needs its interrupt id (HMR_LG_INTERRUPT) to reach the parent, not the sub. Rather than special-case that runtime-specific env name inside the generic dispatch route (a layering smell — the route would have to know LangGraph's contract), the request carries an explicitcontinueParamsmap (wrapper flag--continue-param) merged into the continuation's params alongside the thread id. The reserved-control-param passthrough then delivers the three LangGraph control markers —HMR_LG_THREAD,HMR_LG_RESUME,HMR_LG_INTERRUPT— to the resumed run's env. It is a strict allowlist, not the wholeHMR_prefix (ReservedEnvNames.LANGGRAPH_CONTROL_PARAMS): re-admitting every reserved name would let a callback-API caller setPATH/LD_*in the target's env or spoofHMR_*identity. Every other param reaches the target only through its declaredinputs.
Consequences
- Good: completions are never dropped; the trigger is as reliable as the process status itself.
- Good: exactly-once fire via the PENDING→DISPATCHING claim CAS, identical to dispatch; boot reconciliation (fail orphaned DISPATCHING, re-fire already-terminal PENDING) makes it crash-safe.
- Good: reuses dispatch's config, store, sweep, and reconciliation patterns, so the surface a maintainer must learn is small.
- Bad: the hook runs inline in the executor's
finally; a slow continuation lookup/fire adds latency to job teardown (mitigated — failures are swallowed, the heavy run-start is the same path dispatch already uses). - Bad: a watched result over the 64 KiB cap is not delivered inline; the parent must fetch it out of band.
Later refinement — transient fire failures retry
"The parent must always wake" held for the trigger, but not for fire() itself. A fire could end FAILED without ever starting the target for reasons that had nothing to do with the continuation — the target agent over its cost cap, a full process table, a full run queue — and the only trace was a WARN line. Agents that use continuations as their crash-reporting channel would lose the report of the crash. (Amended 2026-10-09: a full run queue never reaches fire() — maxQueuedRuns is enforced inside the already-launched target run, which ends FAILED like any other run, so only the cost cap, the cooldown and a full process table are retried. The MaxQueuedRunsExceededException arm of isTransient is unreachable; removing it is an open follow-up.)
Those three rejections are now treated as transient: the row goes back to PENDING with the attempt counted (fire_attempts) and the next attempt scheduled (next_attempt_at, doubling from 60 s up to 1 h), and ContinuationSweepService re-fires it on the existing sweep tick. Retries are bounded twice — by an 8-attempt budget and by the continuation's own TTL, which is checked first so a lapsed row expires rather than spending another attempt. (Amended 2026-11-12: the lock-wait bound added by ADR-0104 joins the list — MaxLockWaitersExceededException is classified transient like the run queue. It shares that arm's reachability, though: the lock wait is entered inside the already-launched target run, so the refusal ends that run FAILED with a stored reason and never reaches fire(). The arm keeps the classification exhaustive rather than opening a new retry path.)
Everything else stays permanent and still fails on the first attempt: target agent gone, no longer dispatch-enabled, hop unauthorized, input validation rejected. Retrying those would only burn sweep cycles. failOrphanedDispatching is likewise unchanged — a restart mid-DISPATCHING may already have started the target, so re-firing risks a duplicate run.
Pros and Cons of the Options
Trigger source
- Broadcaster — Good: already fans out process events. Bad:
DROP_OLDESTmeans a completion can be silently dropped; unacceptable for a wake trigger. - Authoritative terminal commit — Good: as reliable as the durable status. Neutral: couples the executor to a post-construction callback.
Inline result
- Always inline — Good: simplest. Bad: an unbounded stdout bloats the resume env and payload.
- Cap + reference fallback — Good: bounded payload. Neutral: large results need an out-of-band fetch.
More Information
Builds on ADR-0020 (durable state in SQLite — continuations table at schema v3), ADR-0021 (ask-and-dispatch — the CAS state model, claim/fire/sweep/boot patterns, and dispatch.enabled gate this mirrors), and ADR-0022 (decoupled dispatch). HTTP endpoints and the homer-dispatch.sh surface that register and inspect continuations ship in a later commit.
Amendment (2026-09-30): settled continuations are pruned
Settled continuations (every status but PENDING and DISPATCHING) are deleted, with their continuation_watches rows, once their created_at is older than server.dispatch.settledRetentionSec, and the list endpoint is paginated. The rule, its created_at anchor and its consequences are recorded once, in ADR-0021's 2026-09-30 amendment.