ADR-0054: Kubernetes workdir uploads without its tar entry
Context and Problem Statement
The kubernetes runner's first upload fails under runAsNonRoot with tar: workdir: Cannot utime: Operation not permitted. The chain:
- Host side,
Workspace.create()createsworkdir/insideprojectRoot;AgentExecutoruploads the whole project root, soworkdir/is a tar entry. PodSpecBuilder.workspaceMounts()mounts the workspace volume a second time at the container's working directory withsubPath("workdir")so the container has a cwd at start.- The kubelet creates the subPath dir after applying
fsGroupto the volume root (kubernetes/kubernetes#81089), ownedroot:<fsGroup>. Verified against a real pod: its mode is cloned from the fsGroup-applied volume root — observed2777(setgid, group/world-writable) — so the pod's non-root user can write into it. Onlyutime/chmodon the directory entry itself requires the owner uid. - GNU tar restores directory mtimes after extraction, so a tar stream carrying a
workdir/directory entry hits thatutimeand fails:EPERM→ tar exit 2 →upload to '/homer/workspace' failed (exit 2): tar: workdir: Cannot utime: Operation not permitted. The same content without that directory entry — files and nested dirs alike — extracts cleanly (exit 0): tar never touches the pre-created directory, only writes into it. - busybox tar (Alpine images) attempts the same restore but silently ignores the
EPERM— verified empirically againstubuntu:24.04(GNU tar, exit 2) andcurlimages/curl:8.21.0(busybox tar 1.37.0, exit 0), both on the real cluster and reproduced in plain docker — so Alpine-based pod images mask the bug entirely, which made it look cluster-dependent rather than image-dependent.
Decision Drivers
- The fix must work on any
runAsNonRootcluster, not depend on the pod image's tar implementation happening to swallow the error. - Kept workspaces (
cleanAfterFinish: false) exist to retainworkdirbetween runs, and the kubernetes runner must syncworkdirthe same way the ssh and local runners do — a caller shouldn't see different round-trip behavior depending on which runner an agent happens to run on. - §12: this is an availability defect — any principal running agents on a non-root kubernetes runner cannot get a first command to run.
Considered Options
- Omit only
workdir's own tar directory entry; upload its contents normally (chosen) — tar never issuesutime/chmodagainst the pre-created directory, but everything inside it uploads like any other workspace content. - Unconditionally exclude
workdirfrom the project-root upload — the originally shipped fix (see Decision Outcome). fsGroupon the pod'ssecurityContext— let the kubelet's group-ownership pass cover the subPath dir too.tar -m(--touch, skip mtime restore) on extraction.- Document
syncExcludes: [workdir]as an operator setting.
Decision Outcome
Chosen option: omit workdir's own tar directory entry from the project-root upload; upload its contents as normal tar entries, scoped to the kubernetes runner.
KubernetesWorkspaceSync.upload gains an omitDirEntries: Set<String> parameter, threaded into addDirContents: a directory whose root-relative entry name is in the set skips putArchiveEntry/closeArchiveEntry for itself but still recurses into its contents, so files and nested directories underneath it upload unaffected. KubernetesRunnerSession passes omitDirEntries = {"workdir"} for the project-root upload (root- anchored by construction — it matches only the top-level entry name, so a workspace-authored nested directory also named workdir still gets its own entry). syncExcludes now passes through unchanged for every upload; workdir no longer needs to be added to it.
This is a revision of the ADR's original decision, which unconditionally excluded workdir from the project-root upload (root-anchored syncExcludes: ["/workdir"] at the session, reusing the root-anchored exclude matching added immediately before this ADR). That shipped first and is now the rejected alternative below.
The other four options are rejected:
- Unconditionally excluding
workdirbreaks the guarantee that a kept workspace'sworkdirround-trips between runs on kubernetes the same way it does on ssh/local, and makes the kubernetes runner behave differently from the others for the samecleanAfterFinish: falsesetting — rejected by the repo owner on 2026-08-26. It was shipped believingworkdir's entire content was unshippable by the reserved-path model thatRunnerConfig.RESERVED_REMOTE_PATHSuses forextraUploadstargets; that reasoning does not hold — that list guards where an operator may point an extra upload, and says nothing about whether the workspace's ownworkdircontent may be synced as part of the normal upload. fsGroupcannot work at all, not just "cannot work reliably": the failure is an ownership mismatch on the directory entry'sutime/chmod(owner uid required), not a group-permission gap — the directory is already group/world-writable (2777) for content, and kubernetes/kubernetes#81089 means the kubelet creates the subPath directory after applyingfsGroupto the volume, so there is no ordering fix available from the pod spec side.tar -mon extraction hides the symptom but drops mtime fidelity for every extracted file on every upload, not just the one directory entry that actually fails — a strictly worse trade for a problem that has a targeted fix.- Operator
syncExcludes: [workdir]is unanchored: it matches the entry name at every directory level, so it would also silently drop any nested directory a repo happens to carry that is namedworkdir. It also leaves the reserved path uploadable by default, requiring every operator on a non-root cluster to discover and set it themselves — and, like unconditional exclusion, it would still break kept-workspace round-tripping.
Consequences
- Good: the kubernetes runner's first upload succeeds under
runAsNonRooton any tar implementation, GNU or busybox — the fix no longer depends on which one silently swallowed the bug. - Good: a kept workspace (
cleanAfterFinish: false) round-tripsworkdiron the kubernetes runner exactly like it does on ssh/local — sync-up uploads the accumulated content back into the next pod's cwd, matching runner parity. - Bad (accepted): the pod's
workdiris no longer guaranteed empty of host content on first upload — it now carries whatever the hostworkdir/contains, same as every other runner. This supersedes the original request's stricter acceptance criterion ("pod workdir stays empty of host content"); the repo owner ruled runner parity and kept-workspace seeding take priority (2026-08-26). - Neutral: the ssh runner is deliberately unchanged. Its remote workdir is a plain directory owned by the ssh user — there is no kubelet subPath pre-creation and no ownership collision, so the failure this ADR fixes does not exist there.
More Information
- Builds on the root-anchored
syncExcludesmatching added immediately before this ADR's first version;omitDirEntriesis a separate, independent mechanism fromsyncExcludes(the latter drops content entirely, the former only skips one directory's own tar entry). - Extends ADR-0043 (sync-mode kubernetes workspaces).