macOS build hosts for mobile builds
A team that builds a mobile app runs Homer on Kubernetes. Its agents build and test the Android app in Linux containers. The iOS app is another matter: Xcode, the simulators and code signing exist only on macOS, and macOS is licensed only on Apple hardware, so no container image can build it. Homer had to run those stages on a Mac.
The runner shape changed twice in about a month. It started as an ssh runner reached through a reverse tunnel. Its workspace teardown then turned out to hold the host for up to 18 minutes after each run. It ended as pull-mode runners on macOS VMs that register themselves. This case study follows that path and what each step needed from Homer.
The setup
Step 1: an ssh runner through a reverse tunnel
Homer's ssh runner always dials out from Homer. The cluster had no route to the Mac's network, but the Mac could reach the cluster's HTTPS ingress. So the Mac kept a reverse tunnel open over that ingress, and the runner pointed at the tunnel's end inside the cluster:
# Homer config (excerpt)
runners:
macos-ssh: !<ssh>
host: mac-tunnel.homer.svc.cluster.local # the tunnel's cluster-side end, not the Mac
port: 2222
user: runner
auth: !<key>
privateKeyPath: /etc/homer/ssh/id_ed25519
knownHostsPath: /etc/homer/ssh/known_hosts # the Mac's own key, also under [mac-tunnel…]:2222
remoteWorkspaceRoot: /Users/runner/homer-workspaces
homerApiUrl: https://homer.example.com # how homer-ask and dispatch reach Homer from the Mac
extraUploads:
- { local: /app/scripts, remote: .homer/scripts, env: SCRIPTS_DIR }
- { local: /app/agents/shared, remote: .homer/shared, env: SHARED_DIR }The tunnel only carries bytes, so the host key Homer checks is still the Mac's. With strictHostKeyChecking on (the default), known_hosts must list that key under the name and port Homer actually dials.
Homer installs nothing on an ssh host (Remote runner requirements). The Mac was provisioned with Xcode, a JDK, bash 5, git, claude, curl and setsid, the last so that Homer can kill the remote process group. Three things in the environment still bit:
- Linux values reached the Mac.
globalEnvheld aPATH, aJAVA_HOMEand aCIflag meant for the Linux containers. Homer sends the run's env with every remote command, so aPATHfromglobalEnvreplaces the login shell's and hides/opt/homebrew/bin. Before Homer 1.24.0 the backend'sHOME=/home/homerwas forwarded too, and on macOS/homeis an autofs mount nothing can be created under. Since 1.24.0 remote runners no longer receiveparentEnvAllowlistkeys, and a per-runnerenvoverrides the rest (Remote runners). - Only the agent's own directory is uploaded. It arrives as
.homer/agent, so{{HMR_AGENT_DIR}}/../shared/…, which works on the local runner, points at nothing on the Mac. The shared scripts and the shared Claude settings travel throughextraUploads, withenvrepointed at the remote copy (ADR-0042). - No drain. An
sshrunner has no drain switch. Host maintenance waited until no SSH session but its own was open, and could not stop Homer from starting a new run meanwhile.
Step 2: the teardown that held the host
Every Mac agent set cleanAfterFinish: true. On an ssh runner, Homer up to 1.24 removed the remote project root over SFTP, one round trip per file, serialized on the session and here through the tunnel. An iOS checkout after dependency resolution is tens of thousands of files. The run stayed WORKING and kept its lock until the sweep ended, 14 to 18 minutes after its last command. A second build that shared the lock waited 650 s for the first one's sweep before its first command ran. On one evening, two tickets spent 58 minutes of a 2 h 52 min run deleting files one SFTP message at a time.
Homer 1.25.0 fixed this (ADR-0067): the ssh runner now removes the project root with one rm -rf over the exec channel and falls back to the SFTP walk only when that fails. Config load also rejects a remoteWorkspaceRoot that is not absolute and normalized, or is /.
The team's workaround on the older version still shaped what came next. The last step of each Mac agent removed the checkout itself in an EXIT trap. Later the agents stopped cloning into the workspace at all and kept one checkout per host at a fixed path outside remoteWorkspaceRoot. Xcode keys its build cache on the project's absolute path, and a workspace path with the process id in it is new every run, so no build had ever found a previous one's cache.
Step 3: pull-mode runners on immutable VMs
Homer 1.26.0 added !<external> runner domains (ADR-0066). The basics, one trust domain and instances that announce their own tags and concurrency, are in Runner pools. For the Mac they removed the tunnel and the SSH key: the Mac runs homer-runner itself and connects out to Homer over HTTPS, and another Mac is another instance with the same token, with no Homer restart.
# Homer config (excerpt)
runners:
macos: !<external>
token: "@SYSENV:HOMER_RUNNER_MACOS_TOKEN"
homerApiUrl: https://homer.example.com # HMR_API_URL for runs on any instance
allowedTags: [macos, ios]
env: # host facts; above agent env for tag-filtered runs
HOME: /Users/runner
CI: "" # globalEnv's value is for the Linux containers
extraUploads: # every instance gets these: no secrets
- { local: /app/scripts, remote: .homer/scripts, env: SCRIPTS_DIR }
- { local: /app/agents/shared, remote: .homer/shared, env: SHARED_DIR }# agents/ios-build/agent.yaml (excerpt)
runnerTags: [ios]
runnerClaimTimeoutSec: 900
maxParallelRuns: 1
lockStrategy:
key: macos-build-host # the same constant in every agent that builds on the Mac
onCollision: WAITThe hosts are macOS VMs (on Tart) cloned from one immutable image. The image carries Xcode, the toolchain and the homer-runner jar from the homer-runner image. At boot a clone reads its domain token and name from a read-only shared directory and starts the runner as a daemon:
HOMER_RUNNER_TOKEN="$(cat /path/to/shared/token)" \
java -jar homer-runner.jar --url https://homer.example.com \
--tags macos,ios --concurrency 1 --root /Users/runner/homer-runner \
--name "$(hostname -s)"A new Xcode, macOS or Homer version is a new image, and VMs are replaced, not updated in place. A clone shows up on the console's /runners page once it has registered. macOS allows at most two macOS VMs on one Mac.
Details that matter
- The lock comes before the claim. Homer takes a run's lock before the run enters the claim queue, so a second build waits on the lock, not on the claim TTL. Lock keys are global across agents, so one constant key makes every Mac agent exclude the others.
runnerClaimTimeoutSecthen covers only the wait for a free instance: a lock-exempt smoke run holding the one slot, orhomer-runnerrestarting. The 300 s default was too short for that. - Domain
envwins over agentenv. For a run that picked its runner byrunnerTags, the serving runner'senvis merged above the agent's when the run is claimed. Put only host facts there, values no agent should override. - Domain
extraUploadsgo to every instance. They are uploaded in full (a domain has nosyncExcludes) into whichever instance claims the run, so the folders must hold no secrets (ExternalRunner fields). - The runner protocol needs its own proxy settings. An instance posts workspace and artifact tars up to
maxTransferBytes(1 GiB by default). nginx's default 1 MB body limit answers413to all but the smallest. The team routes/api/v1/runner/through a separate ingress with a body limit abovemaxTransferBytes, request buffering off, and read and send timeouts of 600 s, above the 30 s claim long-poll and the 300 stransferTimeoutSec. - Build caches need a fixed path.
homer-runnergives each session a fresh<processId>-<random id>directory under--rootand deletes it when the session ends. A cache keyed on the absolute path never hits there. The shared checkout lives outside--root, and the host-wide lock is what makes sharing it safe. globalEnvreaches the Mac, and artifacts are not redacted. The app's build tool printed its whole environment into a log the run kept as an artifact. Every token inglobalEnvended up there in plain text, kept until the workspace was purged. Homer redacts@SYSENV:values in telemetry, not in files a step writes. The fix was a wrapper that drops every variable named like a credential before calling the build tool. Not having the tokens in the run's environment at all is better, see Runs that hold no credentials.- Stop the runner before the VM. An instance that stops polling is dropped after
heartbeatTimeoutSec, and its runs fail. OnSIGTERM,homer-runnerderegisters and lets in-flight runs finish for up toHOMER_RUNNER_DRAIN_TIMEOUTseconds. Send it first, then stop the VM.
What it does not solve
- The domain token is within the runs' reach on macOS. On Linux,
homer-runnermakes its JVM non-dumpable so a run cannot read the token from it (ADR-0078). macOS has noprctl, so the runner logs a warning and goes on. Do not count on the token being out of a run's reach there, and keep the domain for work you trust. - Tags are routing, not isolation. An instance may announce a tag it cannot serve. The run then fails inside a domain you already trust.
- One Mac is a queue. With one instance at a time per VM and a host-wide lock, concurrent tickets wait for each other.
- A shared checkout can carry bad state. A refresh resets tracked files and removes untracked ones, but ignored build output stays. Deleting the checkout between runs is always safe; the next run clones again.
- The
sshpath stays as a fallback, with its tunnel and without a drain.