Bring up the fleet

By the end of this you will have six Debian trixie VMs on your workstation, each running golemd against real apt, real systemd, and a real filesystem.

And every byte of that will have crossed an ssh forward carrying a bearer token, because there is no other way in. This is where you prove that rather than take it on faith.

Commands are nushell, one per line, run from the repo root inside the devenv shell. All of them are safe to re-run.

Before you start

cargo build --workspace
cargo build --release -p golemctl

The workspace build gives fleet its emetc compiler and its target/debug/golemctl. The release build is what puts golemctl on the devenv shell’s PATH (enterShell prepends target/release), which step 7 depends on.

Two waits are worth knowing about before you think something has hung:

• The first fleet up on a machine downloads Debian’s ~340 MB genericcloud image, then waits out cloud-init on each guest. Minutes.
• The first fleet deploy runs nix build .#golemd-static. Minutes again, unless cachix already has it.

Prove the posture

1. Boot six guests.

   fleet up

   Per host you get two lines:

   Booting scaly (ssh 2259, golemd loopback-only)…
     scaly up pid=482913

   Read golemd loopback-only literally. 2259 is scaly’s forwarded ssh port, and for a guest booted this way it is the only port qemu forwards. There is no golemd port in the line because there is no golemd port.

2. Install the daemon.

   fleet deploy

   Building static golemd (musl)…
     binary: /home/you/personal-repos/golem/.fleet/result-golemd-static/bin/golemd
   Deploying golemd to scaly…
     scaly: golemd up {'host': 'scaly', 'latest_revision': 1}

   That latest_revision: 1 is the init revision golemd writes on first boot — an empty scroll, nothing applied yet.

3. Look at the fleet.

   fleet status

   A table, one row per VM: up, golemd reachable, last revision of #1, and dashes under content-id and glyphs because nothing has been applied. Every one of those reachable cells was answered through an ssh forward the harness opened and tore down for that single request.

4. Try the port that used to exist.

   curl -m 5 http://127.0.0.1:8859/status

   curl: (7) Failed to connect to 127.0.0.1:8859 after 0 ms: Could not connect to server

   Exit 7, refused before a packet leaves the machine. Having the fleet up changes nothing. 8859 is the port scaly’s name-derived slot earns (8800 + 59) and it is still recorded in .fleet/state.json, but the only host-to-guest forward qemu creates is ssh — plus whatever you ask for with --publish, which has nothing to do with golemd. The forward that once pointed at 8859 was deleted when the loopback bind made it unreachable by construction, so there is no longer a port to knock on — not a dead one, none.

5. Open a forward yourself, and get refused twice.

   let ssh_opts = [
     "-i" ".fleet/id_ed25519"
     "-p" "2259"
     "-o" "StrictHostKeyChecking=no"
     "-o" "UserKnownHostsFile=/dev/null"
     "-o" "ExitOnForwardFailure=yes"
   ]
   ssh -f -N -L 17501:127.0.0.1:7474 ...$ssh_opts golem@127.0.0.1
   curl -s -w ' [%{http_code}]\n' http://127.0.0.1:17501/status
   curl -s -w ' [%{http_code}]\n' -H 'Authorization: Bearer wrong' http://127.0.0.1:17501/status
   curl -s -w ' [%{http_code}]\n' -H $"Authorization: Bearer (open .fleet/golem-token | str trim)" http://127.0.0.1:17501/status
   pkill -f "17501:127.0.0.1:7474"

   The three curls, in order:

   {"kind":"unauthorized","message":"missing or invalid bearer token — golemd requires Authorization: Bearer <token> (see --auth-token-file)"} [401]
   {"kind":"unauthorized","message":"missing or invalid bearer token — golemd requires Authorization: Bearer <token> (see --auth-token-file)"} [401]
   {"host":"scaly","latest_revision":1} [200]

   Reaching the socket buys nothing. The tunnel is the confidentiality and the host authentication; the token is the authorization; both are required and neither substitutes for the other. See Trust model.

6. Render the inventory.

   fleet inventory

   It prints one path: .fleet/inventory.toml under your checkout. Open it if you like. Each guest is an [hosts.<name>] table carrying ssh, ssh_port, the fleet key and host-checking options as ssh_args, and token_file. No url, because there is nothing to dial.

7. Drive it with no flags and no environment.

   golemctl fleet status

   · kaiju  rev 1  nothing applied
   · manta  rev 1  nothing applied
   · orbit  rev 1  nothing applied
   · scaly  rev 1  nothing applied
   · talos  rev 1  nothing applied
   · zulip  rev 1  nothing applied

   Six lines, alphabetical, and not one flag or environment variable. golemctl looked at --inventory, then $GOLEMCTL_INVENTORY, then ./fleet.toml, then ./.fleet/inventory.toml — and the harness writes its file at the end of that chain deliberately, so being in the repo root is the whole configuration. Each of those six lines is a separate ssh forward, opened and closed, presenting the token that host’s token_file named.

That is the posture proved end to end. Everything from here on rides it.

That third curl leaks the token to /proc

Putting the secret in curl’s argv makes it world-readable for the life of the process. Tolerable for a token that lives in your own checkout and governs throwaway VMs. Against anything real, write the header to a 0600 file and use curl -H @file.

To reach one guest without the inventory, at a bare ssh:// target, see the single-host spelling.

Rotation is one file

Delete .fleet/golem-token, run fleet deploy again, and every guest gets the new secret. Deleting the file generates a new token on the next command, so the redeploy has to reach every guest before anything else will answer. The token survives fleet reset and even reset --purge on purpose, so a rebuilt fleet keeps talking to an inventory you already rendered.

Watch it complain

Point golemctl at a routable address with a token in the environment and it warns on stderr before sending: warning: host X is dialed at Y, which is not loopback — the auth token crosses that network in cleartext; use an ssh:// target to keep it inside the tunnel (ADR 0042). That warning is the only guard. golem will do it; it just will not do it quietly.

Reading the fleet while it works

fleet status
fleet logs scaly -f
fleet ssh scaly

fleet logs and fleet ssh are plain ssh into the guest and are untouched by the auth layer — golemd’s token gates golemd’s HTTP surface, not your shell.

Tear down

fleet down scaly       # stop the VM, keep its disk — a later `up` resumes it
fleet reset            # kill every VM, wipe all per-VM data
fleet reset --purge    # also drop the cached image and keypair

.fleet/golem-token survives all three. What each verb keeps and drops is laid out in The fleet harness.

When it breaks

✗ HOST unreachable: ssh to golem@127.0.0.1 exited 255 before the forward opened — Connection refused The guest is not running, or .fleet/inventory.toml names a port that no longer belongs to it. fleet status says which; re-run fleet inventory after every up, down, or reset, because it is a snapshot of the state file rather than a live view.

✗ HOST unreachable: … on one host while the others answer That guest’s golemd is down or was never deployed. fleet logs HOST still works — it is plain ssh — so read the journal there. fleet deploy --hosts HOST reinstalls and restarts it.

unauthorized — set GOLEM_AUTH_TOKEN or GOLEM_AUTH_TOKEN_FILE (or an inventory host's token_file) to golemd's configured secret The guests hold a different secret than .fleet/golem-token. Almost always a half-finished rotation: run fleet deploy against every host, not just the one that failed.

A verb hangs about ten seconds per host, then reports an ssh error The forward never came up. The message carries ssh’s own stderr. Stopped VM or stale inventory, same fixes as above.

no inventory at … — golemctl looks at --inventory, then $GOLEMCTL_INVENTORY, then ./fleet.toml, then ./.fleet/inventory.toml You are outside the repo root, or you never ran fleet inventory.

error: the fleet token file … is empty Delete it and redeploy everywhere. A truncated secret is refused rather than silently degraded into an unauthenticated request.

Where to next

• A failing unit — break one leaf of a nine-unit tree on purpose and watch its siblings settle anyway.
• A registry on the fleet — a real container registry on a VM, reachable from the others.
• The fleet harness — QEMU, cloud-init, name-derived ports, and what is inside .fleet/.