ADR-0043: Sync-mode kubernetes workspaces — tar-over-exec replaces the shared RWX PVC
Context and Problem Statement
ADR-0003 gave the kubernetes runner a shared RWX PVC workspace model: Homer and every agent pod mount the same claim, and pods see the workspace at the identical absolute path Homer uses. That model has no users today and two hard blockers for the deployments actually wanted:
- Homer outside the cluster (a VM/host bursting runs into k8s via the API server) — PVC sharing is impossible there, period.
- In-cluster without RWX — many storage classes only support RWO; requiring ReadWriteMany gates the runner on storage topology.
Meanwhile the SSH runner already ships the alternative: ADR-0033's lifecycle primitives (createWorkspace / upload / download / cleanWorkspace), driven unconditionally by AgentExecutor and RunnerCmdExecutor, with the kubernetes session today inheriting the no-op defaults. Implementing the primitives on the kubernetes session makes the executors' sync policy work there with zero executor changes. ADR-0042 already reserved extraUploads adoption for exactly this "sync-mode kubernetes runner."
The first real workload driving this is Android/KMP Gradle builds — very large numbers of small files. That profile rules out per-file transfer; a single streamed tar archive per direction is the baseline transport requirement.
Decision Drivers
- Kubernetes runner must work with no shared storage: Homer reaches the pod only through the k8s API (exec over websocket), identically from inside or outside the cluster.
- Replace the PVC model, don't add a second mode — no dead code, no users to migrate incrementally.
- Same operator semantics as the SSH runner:
syncExcludes,extraUploads, env path translation, artifacts-always / workdir-if-kept download policy. - Stop baking the workspace env (secrets included) into the pod spec.
runnerCmdmust work on kubernetes runners (closes the deferred "sync-source story" from ADR-0034).keepPodOnFailuremust become genuinely useful (kept pod retains the workspace) — this also means fixing the never-setfailedflag.
Considered Options
- Keep dual mode — leave the PVC path in place alongside a new sync path, selectable per runner.
- Kubernetes client-java
CopyAPI — the library's built-in tar-over-exec copy helper. - rsync / sync sidecar — a long-lived sidecar container doing delta sync against the workspace.
- Tar-over-exec, implemented in-repo, replacing the PVC path (chosen).
Decision Outcome
Chosen option: "Tar-over-exec, implemented in-repo, replacing the PVC path," because it is the only option that removes the RWX requirement while reusing existing, already-testable plumbing.
- Replace, not add a mode.
workspaceVolumeis removed from config and schema (BREAKING; no users). One model, no dead code. - In-pod workspace: a pod-local
emptyDirmounted at the fixed path/homer/workspace, mirroring the hostprojectRootlayout (workdir/,artifacts/,.homer/agent,.homer/bin,.homer/mcp.json). No configurable root — pod-per-run means no path collisions. - Transport: tar-over-exec via the existing
KubernetesExecClientseam (the same mechanismkubectl cpuses) — upload feeds a host-built tar stream into a pod-sidetar -xzf -; download runs pod-sidetar -czf -and extracts host-side. Gzip always; no compression knob (CPU is trivial next to network transfer, especially with Homer outside the cluster). One stream carries any number of small files — the best case for the Gradle profile, and it works identically from inside or outside the cluster. - Client
CopyAPI rejected as the implementation vehicle: it is the same tar-over-exec mechanism underneath, minus exclude filtering, overlay semantics, and a test seam — adopting it would mean re-implementing the parts that matter on top of it anyway. Tar framing is implemented in-repo instead (host-side via commons-compress). - Pod spec stops carrying the workspace env. Commands already receive the full merged env per-exec (
env KEY=VAL … argv); the baked copy's only real effects were leaking agent secrets intokubectl get pod -o yamland serving manualkubectl execshells. Removing it deletes the "Secret exposure in the pod spec" surface SERVER.md documents. syncExcludesandextraUploadsadopted on the kubernetes runner with the same shape and semantics as the SSH runner (ADR-0042: "sync-mode kubernetes runner picks up the same field unchanged").cleanWorkspace()stays a no-op on kubernetes — theemptyDirdies with the pod. A kept-for-debug pod deliberately retains its workspace; that retention is the debugging payoff sync mode buys.markFailed()added toRunnerSession(default no-op;AgentExecutorcalls it on failure paths before close) soKubernetesRunnerSession.markFailed()can set thefailedflag thatkeepPodOnFailurehas always depended on but that was never set — the knob was previously inert.- Image requirement:
tar+gzip(busybox suffices) — the same requirementkubectl cpalready imposes, effectively universal in build images.
Consequences
- Good: kubernetes runner works from outside the cluster and in-cluster without RWX storage — the two blockers this ADR exists to remove.
- Good:
runnerCmdbecomes functional on kubernetes runners with no special-casing —RunnerCmdExecutordrives the primitives generically. - Good:
keepPodOnFailurebecomes genuinely useful oncemarkFailed()wires the flag it always relied on. - Good: build caches (e.g. Gradle home) remain expressible via the existing generic
volumesfield — no new caching feature needed. - Bad: BREAKING config change —
workspaceVolumeis removed; migration is to delete the field from runner config. - Bad: live artifact tailing mid-run is gone for kubernetes runners (artifacts now appear host-side only at download) — this brings kubernetes to parity with the SSH runner, which never had it.
Pros and Cons of the Options
Keep dual mode (rejected)
- Good: zero migration for any hypothetical existing PVC-mode user.
- Bad: the PVC path has no users today — keeping it is dead code with an ongoing maintenance and test-matrix cost for no benefit.
Kubernetes client-java Copy API (rejected)
- Good: less code to own; library-maintained.
- Bad: no exclude filtering, no overlay control, no test seam — the parts that make sync match SSH semantics would be re-implemented on top of it regardless, leaving two tar code paths instead of one.
rsync / sync sidecar (rejected)
- Good: would give incremental/delta sync on subsequent syncs.
- Bad: pods are per-run and start empty — there is nothing to delta against. All the cost of a sidecar (extra container, lifecycle, RBAC) for no benefit in this lifecycle.
Security considerations
- Pod spec no longer carries agent secrets —
kubectl get podin the runner namespace stops being a secret-disclosure channel, and neither does etcd or any informer watching pod objects. Per-exec env still travels over the API-server websocket session (TLS), not into a stored object. The residual surface: the exec client passes the full merged env as repeatedenv KEY=VAL …arguments in the exec request'scommand=query parameters, which are part of the request URI — and Kubernetes audit events recordrequestURIat every audit level, including the commonMetadatadefault, so any cluster auditingpods/execat all captures the full per-exec env, not only atRequestResponse. Operators should scope the audit policy forpods/execaccordingly. RBAC onpods/execremains the boundary that matters. - Workspace content now transits the API server instead of a storage backend — no new trust party (the API server already brokers exec).
- Kept-for-debug pods retain workspace content, including any secrets agents materialized into files; the existing
keepPodTtlSecondsbound is unchanged.
Relationship to ADR-0003
This ADR supersedes the workspace-sharing portion of ADR-0003 — specifically its "Workspaces are shared via a single RWX PVC..." decision and the consequence "live tailing keeps working because pod stdout/stderr are streamed to files on the shared PVC." The rest of ADR-0003 (the Runner abstraction itself: one pod per agent run, session.shell as the backend-agnostic seam, HMR_API_URL contributed by the runner session) is unaffected and stays in force; ADR-0003's status is left accepted rather than flipped to superseded, since only that one sub-decision is replaced.
More Information
- Related: ADR-0003 (runner abstraction; workspace-sharing portion superseded here), ADR-0033 (lifecycle primitives this ADR implements on the kubernetes session), ADR-0034 (
runnerCmd, whose kubernetes sync-source story this ADR closes), ADR-0042 (extraUploads, adopted unchanged by the kubernetes runner here).