A registry on the fleet

examples/registry/ runs a real Podman registry on a golem-managed box, and uses it from another. It is the first half of golem’s dogfood loop: one VM hosts a registry, another pushes to it — no shared L2 segment, just golem and QEMU’s host gateway.

Start from Bring up the fleet; this needs the same harness and the same ssh-plus-token posture. Commands are nushell, from the repo root.

The registry, as one Workload

registry.emet is a single Quadlet.Workload — the docker.io registry:2 image on :5000, backed by a named volume, internally exposed:

import Quadlet exposing
  ( image, tcp, Restart(..), Access(..), Relabel(..)
  , fromVolume, Expose(..), Workload(..), workloadGlyphs )

registry : Workload
registry =
  Workload
    { name = "registry"
    , image = image "docker.io/library" "registry" "2"
    , env = []
    , ports = [ tcp 5000 5000 ]
    , volumes =
        [ fromVolume "golem-registry-data" "/var/lib/registry" ReadWrite Private ]
    , restart = Always
    , expose = Internal
    }

main : List Scroll
main =
  [ scroll
      { name = "registry"
      , notifies = [ "golem-nftables.service" ]
      , glyphs = workloadGlyphs registry
      }
  ]

It lowers to eleven glyphs — the podman package, the golem-registry-data.volume unit, the .container quadlet, its service, the nftables base, and the drop-in that opens :5000 to the internal network:

main : List Scroll
planned scrolls (1):
  scroll `kaiju` (11 glyphs):
    * ensure apt package `podman` installed
    * ensure file `/etc/containers/systemd/golem-registry-data.volume` (mode 0644)
    * ensure file `/etc/containers/systemd/registry.container` (mode 0600)
    * enable + start systemd unit `registry.service`
    * ensure apt package `nftables` installed
    * ensure directory `/etc/nftables.d` (mode 0755)
    * ensure file `/etc/golem-nftables.conf` (mode 0755)
    * ensure file `/etc/nftables.d/00-base.nft` (mode 0644)
    * ensure file `/etc/systemd/system/golem-nftables.service` (mode 0644)
    * enable + start systemd unit `golem-nftables.service`
    * ensure file `/etc/nftables.d/registry-5000.nft` (mode 0644)
    ↻ kaiju notifies golem-nftables.service

Boot the three boxes

The registry is only useful with something to push to it and something to serve from it, so bring up all three now. --publish adds a host-to-guest forward, which is unrelated to golemd — golemd is still loopback-only on every guest.

1. Boot and deploy. The web box publishes on 8081 only because 8080 was busy here; any free port works.

   fleet up --hosts kaiju,talos,remora --publish kaiju=5000:5000 --publish remora=8081:80
   fleet deploy --hosts kaiju,talos,remora

2. Apply the registry and builder scrolls. On cold guests this is the long step — apt update, installing podman, pulling registry:2. Many minutes, streaming progress the whole time; a quiet terminal is work, not a hang.

   fleet apply examples/registry/registry.emet --hosts kaiju
   fleet apply examples/website/builder.emet --hosts talos

   The builder scroll is two glyphs: podman, and a registries.conf.d fragment marking 10.0.2.2:5000 insecure so podman will push to it over plain HTTP.

3. Confirm the registry is serving. Its :5000 is published to your workstation:

   curl -s http://127.0.0.1:5000/v2/
   fleet ssh kaiju -- systemctl status registry.service

   An empty registry answers /v2/ with an empty JSON object, and systemctl reports registry.service as active (running):

   {}

When it does not come up

curl: (7) Failed to connect to 127.0.0.1:5000 The guest was booted without --publish kaiju=5000:5000. Forwards are recorded per VM at boot, so re-run fleet up --hosts kaiju --publish registry=5000:5000 — a resume can add forwards the stopped guest lacked.

The apply reports the registry leaf failed Read the pull error in the forensics block under it, then fleet ssh kaiju -- journalctl -u registry.service -n 50 for what podman said. No sudo needed: cloud-init put golem in systemd-journal.

registry.service is active (running) but the catalog curl hangs The container is up and the ruleset that opens :5000 is not. fleet ssh kaiju -- systemctl status golem-nftables.service says whether the reload the scroll’s notifies asked for actually ran.

How the builder reaches the registry

Each guest runs behind user-mode (SLIRP) networking and is isolated from the others — only ssh is forwarded in by default. Two facts bridge them:

• up --publish kaiju=5000:5000 binds your 127.0.0.1:5000 to the registry guest’s :5000.
• In SLIRP, every guest reaches your workstation at 10.0.2.2.

So any guest reaches the registry’s published port at 10.0.2.2:5000 — the connection lands on your loopback, which QEMU forwards into the registry guest. That rendezvous is the whole reason the builder scroll names 10.0.2.2:5000:

registryEndpoint : String
registryEndpoint = "10.0.2.2:5000"

builderGlyphs : List Glyph
builderGlyphs =
  [ aptPackage { name = "podman" }
  , file
      { path = "/etc/containers/registries.conf.d/golem-registry.conf"
      , contents = insecureConfContents      -- marks 10.0.2.2:5000 insecure
      , mode = "0644"
      }
  ]

The mechanics are in The fleet harness.

Insecure on purpose

The drop-in marks the registry insecure so podman pushes over plain HTTP — fine for an ephemeral local fleet, not for anything real. Registry auth and TLS are their own concern, out of scope for the dogfood loop.

Push something to it

fleet ssh talos -- "sudo podman pull docker.io/library/hello-world:latest && sudo podman tag docker.io/library/hello-world:latest 10.0.2.2:5000/hello:latest && sudo podman push 10.0.2.2:5000/hello:latest"
curl -s http://127.0.0.1:5000/v2/_catalog

{"repositories":["hello"]}

The quoted && chain runs in the guest’s bash over ssh, so nushell never sees it.

One golem-hosted registry, reachable from a machine that shares no network segment with it.

Two clients instead of one

examples/registry/clients.emet provisions a builder and a puller with the same two glyphs, if you want to watch a third box pull what the second pushed: fleet up --hosts orbit, then fleet apply examples/registry/clients.emet --hosts talos,puller.

Explore from here

• Watch the diff. Re-apply registry.emet after bumping the image tag — only the .container glyph’s content id changes, so only that glyph is replaced.
• Reverse it. Apply a registry scroll with an empty glyphs list and watch golem stop the service, remove the quadlet and .volume units, close the firewall port, and remove podman — each undone because golem recorded it.
• Inspect the volume. fleet ssh kaiju -- sudo ls /var/lib/registry — the named volume that survives a container restart.

Where to next

• The website loop — build the docs into an image, push it to this registry, and serve it from the web box.
• The Quadlet library — every field of the Workload this registry is.