A tour of the lichess fleet

The capstone example is the lichess fleet: six hosts running dozens of containerized services behind nginx ingress. It lives in examples/lichess/ and compiles as-is. It is also the clearest place to see golem’s three layers at once:

• (a) the four glyphs — aptPackage, systemdService, file, lineInFile — the only kinds the agent enacts.
• (b) Quadlet — golem’s shipped, strongly-typed Podman-quadlet library. It lowers to (a).
• (c) Lichess.emet — this fleet’s own ergonomic helpers, built on (b).

The example is three files: Fleet.emet (shared facts), Lichess.emet (the layer-c helpers), and fleet.emet (the six hosts). Everything higher-level lowers to the four glyphs — no fifth kind.

Layer b: the shipped Quadlet library

A container on a Debian box is a Podman quadlet — a .container unit systemd runs, plus the .volume units it mounts. Quadlet models those as typed values. The image is not a string; it is a registry, a name, and a tag:

type Ref = Tag String | Digest String

type Image = Image { registry : String, name : String, ref : Ref }

image : String -> String -> String -> Image
image registry name tag = Image { registry = registry, name = name, ref = Tag tag }

A published port carries its protocol; the restart policy is a closed enum; a volume mount is a sum, because a named volume and a host path lower differently:

type Port = Port { host : Int, container : Int, proto : Proto }

type Restart = Always | OnFailure | No

type Mount
  = FromVolume { volume : String, at : String, access : Access, relabel : Relabel }
  | FromHost   { source : String, at : String, access : Access, relabel : Relabel }

The ergonomic surface is Workload, which folds all of this — image, env, ports, volumes, restart, and firewall exposure — into one record, and workloadGlyphs lowers it to the four glyphs. The Quadlet reference documents every type; the point here is that this library is shipped and typed, not hand-rolled per fleet.

Layer c: the fleet’s own helpers

Lichess.emet is not part of the library — it is this fleet’s shorthand for the shapes it repeats, written on top of Workload. dockerImage pins the registry; service is a workload with one port, opened to the internal network:

import Quadlet exposing
  ( Image, image, tcp, Restart(..), Expose(..), Workload(..), workloadGlyphs )

dockerImage : String -> String -> Image
dockerImage name tag = image "docker.io" name tag

service : String -> Image -> Int -> List Glyph
service name img port =
  workloadGlyphs
    (Workload
      { name = name
      , image = img
      , env = []
      , ports = [ tcp port port ]
      , volumes = []
      , restart = Always
      , expose = Internal
      })

workload is the same with no port and expose = Unexposed. These are thin — they just pin the fields this fleet always sets the same way and delegate to the shipped workloadGlyphs. A different fleet would grow different helpers over the same library.

service is shown simplified. Lichess.emet spells it expose = Unexposed followed by an explicit List.concat of nftablesBase and its own nftablesDropIn — the same firewall glyphs Internal would have derived, assembled by hand from the pieces ingress needs anyway. The two spellings compile to a byte-identical manifest.

Ingress stays hand-rolled

ingress is the one shape that does not go through Workload, on purpose. A reverse proxy is a different container’s config — the nginx front door in front of a service, not a property of the service itself — so it is its own set of glyphs:

import Nftables exposing (nftablesBase, nftablesDropIn)

ingress : String -> String -> Int -> List Glyph
ingress name upstream port =
  List.concat
    [ nftablesBase
    , [ aptPackage { name = "nginx" }
      , file { path = nginxSitePath name, contents = nginxSite name upstream, mode = "0644" }
      , nftablesDropIn "ingress-${name}" [ publicAcceptRule name port ]
      , systemdService { unit = "nginx.service" }
      ]
    ]

The nginx package, the site-config file, one firewall drop-in, and the nginx unit — with Nftables.nftablesBase ahead of them. Both bodies are rendered by String.join at compile time.

nftablesDropIn writes one complete file per opening, at /etc/nftables.d/ingress-lichess.org.nft, holding a whole table inet golem block with one accept rule in it. Nothing appends to a shared file. nftables merges table and chain blocks additively across a single atomic nft -f load, so the merge is the consumer’s job, not the author’s — and removing an ingress deletes its one file rather than performing surgery on a file other ingresses still depend on.

nftablesBase is the rest of that arrangement: the nftables package, the /etc/nftables.d directory, an entrypoint conf that adds and flushes table inet golem and then includes the drop-in glob, a 00-base drop-in carrying the hooked input chain, and the oneshot golem-nftables.service that loads it. golem owns that one table and touches no other — the flush is scoped to it, so a reload never disturbs the tables podman keeps next door.

One file per glyph, never a shared file

golem composes config by dropping complete files into a directory it also owns, never by editing lines into a file several units share (ADR 0041). Line composition has no owner, no declared order, and no clean removal. lineInFile stays in the language for the other case — editing a file golem did not author. See The four glyphs.

Order matters, because it is the only ordering there is

nftablesBase comes first in the list, and that is load-bearing: it carries the /etc/nftables.d directory glyph, and a scroll’s glyphs enact in source order with no dependency graph behind them. Every unit that writes a drop-in repeats the base; identical contributions collapse by content id, so the work happens once.

Every host that opens a port then says notifies = [ "golem-nftables.service" ] on its scroll, so a changed drop-in earns a reload of the ruleset at the end of the apply. scaly is the one that does not, because it opens none. The base chain’s policy is drop, which makes that allowlist real: a port nothing declares is closed. Its standing accepts are the ones that keep a box reachable — established traffic, loopback, icmp, dhcp, ssh, and golemd’s own 7474. That last one is vestigial: the agent binds loopback and the next apply arrives over SSH, so ssh is what keeps the box reachable to golem at all.

The fleet

fleet.emet imports the layer-c helpers and defines each host as a Scroll — a List.concat of helper calls:

module Main exposing (..)

import Lichess exposing
  ( dockerImage
  , ghcrImage
  , workload
  , service
  , ingress
  , host
  )

scaly : Scroll
scaly = host "scaly" (workload "fishnet" (dockerImage "niklasf/fishnet" "latest"))

manta : Scroll
manta =
  scroll
    { name = "manta"
    , notifies = [ "golem-nftables.service" ]
    , glyphs =
        List.concat
          [ workload "leroyjenkins-lila" (ghcrImage "lichess-org/leroyjenkins" "latest")
          , workload "leroyjenkins-nginx" (ghcrImage "lichess-org/leroyjenkins" "latest")
          , service "lila" (ghcrImage "lichess-org/lila" "latest") 9663
          , service "redis-server" (dockerImage "redis" "7") 6379
          , ingress "lichess.org" "lila" 443
          , ingress "lichess1.org" "lila" 443
          ]
    }

main : List Scroll
main = [ scaly, manta, orbit, talos, kaiju, remora ]

Six hosts, one main. scaly runs a single networkless workload; manta fronts lila with two ingresses; kaiju is one mongo service; talos runs the monitoring stack. Every host reads as the shapes you care about, and every shape lowers to the four glyphs.

The two hosts are spelled differently on purpose. host builds a flat leaf scroll with no notifies, which suits scaly because it opens no ports. Any host carrying a service or an ingress writes its own scroll with notifies = [ "golem-nftables.service" ], so the drop-ins it adds are actually loaded.

scaly and talos are also the live-tested pair: both sit on the public niklasf/fishnet image. manta and orbit are the two hosts you cannot run for real — their workloads and services name ghcr.io/lichess-org refs that are not publicly pullable. This is an illustrative topology, not a deployable one.

Cross-host facts are values, not templates

Fleet.emet holds the shared facts every host reads — the internal CIDR, an address:port helper — as ordinary Emet values:

module Fleet exposing (internalNetwork, endpoint)

internalNetwork : String
internalNetwork = "10.0.0.0/8"

service’s firewall opening reads internalNetwork as imported data, checked by the type system — not a placeholder a resolver hopes to fill in later.

Lichess and Fleet both live in examples/lichess/ beside the entry module, and resolution is entry-directory-first, so those copies win — the Fleet in lib/ is shadowed and never loaded. What the repo-root emet.json search path supplies here is Nftables and Quadlet, which Lichess imports and no file in the entry directory provides (ADR 0024).

Compile and inspect

# Eyeball one host's plan — real glyph counts, real keys.
cargo run -q -p emet -- build examples/lichess/fleet.emet --text

scaly’s single workload lowers to three glyphs:

main : List Scroll
planned scrolls (6):
  scroll `scaly` (3 glyphs):
    * ensure apt package `podman` installed
    * ensure file `/etc/containers/systemd/fishnet.container` (mode 0600)
    * enable + start systemd unit `fishnet.service`

manta — the same helpers, but two workloads, two services, and two ingresses — lowers to forty-six. Both blocks are the compiler’s own output, cut from one run over examples/lichess/fleet.emet:

main : List Scroll
planned scrolls (6):
  scroll `manta` (46 glyphs):
    * ensure apt package `podman` installed
    * ensure file `/etc/containers/systemd/leroyjenkins-lila.container` (mode 0600)
    * enable + start systemd unit `leroyjenkins-lila.service`
    * ensure apt package `podman` installed
    * ensure file `/etc/containers/systemd/leroyjenkins-nginx.container` (mode 0600)
    * enable + start systemd unit `leroyjenkins-nginx.service`
    * ensure apt package `podman` installed
    * ensure file `/etc/containers/systemd/lila.container` (mode 0600)
    * enable + start systemd unit `lila.service`
    * ensure apt package `nftables` installed
    * ensure directory `/etc/nftables.d` (mode 0755)
    * ensure file `/etc/golem-nftables.conf` (mode 0755)
    * ensure file `/etc/nftables.d/00-base.nft` (mode 0644)
    * ensure file `/etc/systemd/system/golem-nftables.service` (mode 0644)
    * enable + start systemd unit `golem-nftables.service`
    * ensure file `/etc/nftables.d/lila-9663.nft` (mode 0644)
    * ensure apt package `podman` installed
    * ensure file `/etc/containers/systemd/redis-server.container` (mode 0600)
    * enable + start systemd unit `redis-server.service`
    * ensure apt package `nftables` installed
    * ensure directory `/etc/nftables.d` (mode 0755)
    * ensure file `/etc/golem-nftables.conf` (mode 0755)
    * ensure file `/etc/nftables.d/00-base.nft` (mode 0644)
    * ensure file `/etc/systemd/system/golem-nftables.service` (mode 0644)
    * enable + start systemd unit `golem-nftables.service`
    * ensure file `/etc/nftables.d/redis-server-6379.nft` (mode 0644)
    * ensure apt package `nftables` installed
    * ensure directory `/etc/nftables.d` (mode 0755)
    * ensure file `/etc/golem-nftables.conf` (mode 0755)
    * ensure file `/etc/nftables.d/00-base.nft` (mode 0644)
    * ensure file `/etc/systemd/system/golem-nftables.service` (mode 0644)
    * enable + start systemd unit `golem-nftables.service`
    * ensure apt package `nginx` installed
    * ensure file `/etc/nginx/sites-enabled/lichess.org.conf` (mode 0644)
    * ensure file `/etc/nftables.d/ingress-lichess.org.nft` (mode 0644)
    * enable + start systemd unit `nginx.service`
    * ensure apt package `nftables` installed
    * ensure directory `/etc/nftables.d` (mode 0755)
    * ensure file `/etc/golem-nftables.conf` (mode 0755)
    * ensure file `/etc/nftables.d/00-base.nft` (mode 0644)
    * ensure file `/etc/systemd/system/golem-nftables.service` (mode 0644)
    * enable + start systemd unit `golem-nftables.service`
    * ensure apt package `nginx` installed
    * ensure file `/etc/nginx/sites-enabled/lichess1.org.conf` (mode 0644)
    * ensure file `/etc/nftables.d/ingress-lichess1.org.nft` (mode 0644)
    * enable + start systemd unit `nginx.service`
    ↻ manta notifies golem-nftables.service

The count is high because the nftables base repeats behind every service and ingress that opens a port. On the host it is one set of files: the six base glyphs share one key and one content id each, so the first unit that reaches them does the work and the rest are credited without touching the host. The trailing ↻ line is the scroll’s notifies — the reload that will run once, at the end of the apply, after the last drop-in lands.

The manifest carries all six scrolls; each golemd selects its own by --host name and ignores the rest — see Manifest format. To run the whole thing against ephemeral VMs, see Bring up the fleet.

What you just saw

The fleet is authored in the vocabulary of the problem — workloads, services, ingresses, an internal network — and none of that vocabulary reaches the agent. It all lowers to aptPackage, systemdService, and the filesystem glyph, checked by Emet’s type system on the way down. The fleet reaches for no lineInFile at all: every file it writes is one golem owns outright.

• The library is shipped and typed. Image, Port, Mount, Workload come from Quadlet — you fill in records, not Image= strings.
• Helpers are yours. dockerImage, service, ingress are this fleet’s layer-c shortcuts, not engine features.
• Ingress is not a workload. A reverse proxy is a different container’s config, so it stays hand-rolled glyphs — the library models runtimes, not proxies.
• Upgrade and removal are content-addressed. Bump an image tag, recompile, re-apply — only the changed glyphs are replaced. See Reversible reconcile.

Where to next

• Every Quadlet type in detail: The Quadlet library
• Run this on real boxes: Bring up the fleet
• The mechanism underneath: Reversible reconcile