docs: promote vm run + image catalog as the happy path

Lead the README with `banger vm run` (one command, auto-pull default
image + kernel from the catalogs), move `image register` / `image
build` / OCI-pull to a "power-user flows" section. Golden-image
content from customize.sh moves to the golden-image Dockerfile story.

New `docs/image-catalog.md` mirrors `docs/kernel-catalog.md` — the
bundle format, content-addressed filenames, publish flow, trust
model, R2 hosting. Cross-links with oci-import.md.

`docs/oci-import.md` refactored to document the OCI-pull path as the
fallthrough for arbitrary registry refs (it's the secondary path now
that the catalog covers the headline debian-bookworm case). Phase A
caveats removed — ownership fixup, agent injection, and first-boot
sshd install all landed.

AGENTS.md: promotes `vm run` as the smoke-test primitive, notes the
default-image auto-pull behaviour, and points at both catalog docs.

README shrinks 330 → 198 lines, mostly by removing the experimental
void/alpine sections (those flows still work as advanced scripts but
the README no longer advertises them).

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
This commit is contained in:
Thales Maciel 2026-04-18 15:33:30 -03:00
parent 75baf2e415
commit 8029b2e1bc
No known key found for this signature in database
GPG key ID: 33112E6833C34679
4 changed files with 404 additions and 450 deletions

123
docs/image-catalog.md Normal file
View file

@ -0,0 +1,123 @@
# Image catalog
The image catalog ships pre-built banger rootfs bundles so users don't
have to register or build anything. It's the fast path behind
`banger vm run` (auto-pull) and `banger image pull <name>`. The
catalog is embedded into the banger binary and updated each release.
End-user flow:
```bash
banger image pull debian-bookworm # explicit
banger vm run --name sandbox # implicit (auto-pulls)
```
## Architecture
Two parts — the same shape as the kernel catalog:
1. **`internal/imagecat/catalog.json`** — JSON manifest embedded into
the banger binary via `go:embed`. Each entry: name, distro, arch,
kernel_ref (a `kernelcat` entry name), tarball URL, tarball
sha256, size.
2. **Tarballs at `https://images.thaloco.com/`** — Cloudflare R2
bucket `banger-images`, fronted by a public custom domain. Each
tarball is `<name>-<arch>-<sha256-prefix>.tar.zst` (content-
addressed filename so CDN edge cache can never serve stale bytes
for the URL the catalog points at). Contents at the archive root:
`rootfs.ext4` (finalized: flattened + ownership-fixed + agent-
injected at build time) and `manifest.json`.
The `banger image pull` bundle path streams the tarball, verifies
sha256 against the catalog entry, extracts both files into a staging
dir, resolves the kernel via `kernel_ref` (auto-pulling from
`kernelcat` if the user hasn't pulled it yet), stages boot artifacts
alongside, and registers the result as a managed image.
The same `image pull` command transparently falls through to the
existing OCI-pull path when `<name>` doesn't match a catalog entry —
see [`docs/oci-import.md`](oci-import.md).
## Adding or updating an entry
The repo has no CI for bundle publishing yet. Catalog updates are
manual.
```bash
# 1. Build the bundle + upload + patch catalog.json in one shot.
scripts/publish-golden-image.sh
# 2. Review and commit the catalog change.
git diff -- internal/imagecat/catalog.json
git add internal/imagecat/catalog.json
git commit -m 'imagecat: publish debian-bookworm'
# 3. Rebuild so the new catalog is embedded.
make build
```
`scripts/publish-golden-image.sh` wraps `scripts/make-golden-bundle.sh`
(which runs `docker build` on `images/golden/Dockerfile` then pipes
`docker export` into `banger internal make-bundle`), computes the
bundle's sha256, uses the first 12 hex chars as a cache-busting
filename suffix, uploads via `rclone` to R2, HEAD-checks the public
URL, and patches `internal/imagecat/catalog.json`.
Environment overrides if the defaults need to change:
`RCLONE_REMOTE`, `RCLONE_BUCKET`, `BASE_URL`.
`--skip-upload` builds the bundle into `dist/` and stops — useful for
local testing without touching R2 or the catalog.
## Bundle format
A bundle is a tar+zstd archive with exactly two entries at the root:
```
rootfs.ext4 # finalized banger rootfs
manifest.json # {name, distro, arch, kernel_ref, description}
```
`rootfs.ext4` is fully prepared at build time: ownership fixed via
`debugfs sif`, banger guest agents (vsock agent, network bootstrap,
first-boot unit) already injected and enabled in
`multi-user.target.wants`. The pull path only has to place the file
and register the image — no mkfs, no ownership pass, no injection on
the daemon host.
## Removing an entry
1. Remove the entry from `internal/imagecat/catalog.json` and commit.
2. Delete the tarball from R2:
`rclone delete banger-images:banger-images/<name>-<arch>-<hash>.tar.zst`.
3. Rebuild banger.
Already-pulled local images are not invalidated — users keep using
them until they run `banger image delete <name>`.
## Versioning conventions
- **Entry names**: `<distro>-<release>` (e.g. `debian-bookworm`).
Per-release names make it trivial to publish `debian-trixie`
alongside without collisions.
- **Content-addressed filenames**: the `-<sha256-prefix>` suffix is
mandatory (set by `publish-golden-image.sh`). Never reuse a URL for
different bytes.
- **Architecture**: `x86_64` only today. The `arch` field is additive
— adding `arm64` is a config change, not a schema change.
## Trust model
Same as the kernel catalog: the embedded `catalog.json` carries each
bundle's sha256, and `imagecat.Fetch` rejects any download whose hash
doesn't match. This protects against transport corruption and against
an attacker swapping an R2 object without landing a commit in the
banger repo. GPG/sigstore signing is deferred until banger is public
and the threat model justifies the operational overhead.
## Hosting
Tarballs live in Cloudflare R2 (bucket `banger-images`), served at
`images.thaloco.com`. The bucket is publicly readable; writes require
the R2 API token configured on the `banger-images` rclone remote.

View file

@ -1,193 +1,153 @@
# OCI import (`banger image pull`)
`banger image pull <oci-ref>` downloads a container image from any
OCI-compatible registry (Docker Hub, GHCR, quay.io, self-hosted, …),
flattens its layers into an ext4 rootfs, and registers the result as
a managed banger image.
`banger image pull` has two paths. The primary one — catalog bundle —
is documented in [`docs/image-catalog.md`](image-catalog.md). This
doc covers the fallthrough: OCI-registry pull for arbitrary container
images.
Paired with the kernel catalog, this dissolves the "where do I get a
rootfs" bottleneck for most users — any distro that ships an official
container image can now boot (eventually) as a banger VM.
## When to use it
Use the OCI path when you need a distro or image that isn't in the
catalog. The catalog covers the common happy path
(`debian-bookworm`); anything else (`alpine`, `fedora`, `ubuntu`,
custom corporate images) goes through OCI pull.
```bash
banger kernel pull void-6.12
banger image pull docker.io/library/debian:bookworm --kernel-ref void-6.12
banger image list # debian-bookworm appears, Managed=true
banger image pull docker.io/library/alpine:3.20 --kernel-ref generic-6.12
banger image pull ghcr.io/myorg/devimg:v2 --kernel-ref generic-6.12
```
`banger image pull` dispatches based on the reference:
- `banger image pull debian-bookworm` → catalog (fast path).
- `banger image pull docker.io/library/foo:bar` → OCI (anything not
in the catalog).
## What works
- Pulling any public OCI image that exposes a `linux/amd64` manifest.
- Any public OCI image that exposes a `linux/amd64` manifest.
- Correct layer replay with whiteout semantics (`.wh.*` deletes,
`.wh..wh..opq` opaque-dir markers).
- Path-traversal and relative-symlink-escape protection.
- Content-aware default sizing (`content × 1.25`, floor 1 GiB).
- Layer caching on disk, keyed by blob SHA256.
- **File ownership preservation.** Tar-header uid/gid/mode is captured
during flatten and applied to the resulting ext4 via a `debugfs`
pass, so setuid binaries (`sudo`, `passwd`) and root-owned config
files (`/etc/shadow`, `/etc/sudoers`) end up correctly owned.
- **Banger guest agents pre-injected.** The pulled ext4 ships with
`/usr/local/bin/banger-vsock-agent`, `banger-network.service`, and
`banger-vsock-agent.service` already in place and enabled.
- **First-boot sshd install.** A one-shot systemd service installs
`openssh-server` via the guest's package manager on first boot —
apt-get / apk / dnf / pacman / zypper dispatch based on
`/etc/os-release`. Subsequent boots skip the install.
- Piping pulled images into the existing `banger image build
--from-image` flow.
- Content-aware default sizing (`content × 1.5`, floor 1 GiB).
- Layer caching on disk, keyed by blob sha256.
- **Ownership preservation** — tar-header uid/gid/mode captured
during flatten, applied to the ext4 via a `debugfs` pass, so
setuid binaries (`sudo`, `passwd`) and root-owned config
(`/etc/shadow`, `/etc/sudoers`) end up correctly owned.
- **Pre-injected banger agents** — the pulled ext4 ships with
`banger-vsock-agent`, `banger-network.service`, and the
`banger-first-boot` unit already enabled.
- **First-boot sshd install** — a one-shot systemd service installs
`openssh-server` via the guest's package manager on first boot.
Dispatches on `/etc/os-release``apt-get` / `apk` / `dnf` /
`pacman` / `zypper`. Subsequent boots skip the install.
- Composition with `image build --from-image`.
## What doesn't yet work
- **Private registries**. Auth is not implemented; anonymous pulls
only. Docker Hub, GHCR (public), quay.io (public), etc. all work.
- **Non-`linux/amd64` platforms**. The kernel catalog is x86_64-only,
so pulled rootfses match. `arm64` is additive in the schema; wire-
up lands when a user needs it.
- **Non-systemd distros.** The injected units assume systemd as PID 1.
Alpine ≥3.20 ships systemd; older alpine + void + busybox-init
images won't honour the banger-network / banger-first-boot units.
- **First boot needs network access.** The provisioning step reaches
out to the distro's package repo to install openssh-server. VMs
without NAT or without the bridge reaching the internet will time
out on first boot. The marker file stays in place so a later boot
retries.
- **Private registries**. Anonymous pulls only. Docker Hub, GHCR
(public), quay.io (public) all work. Adding auth via
`authn.DefaultKeychain` (from `go-containerregistry`) is a cheap
follow-up when someone needs it.
- **Non-`linux/amd64`**. The kernel catalog is x86_64-only, so pulled
rootfses match. `arm64` is additive in the schema.
- **Non-systemd rootfses**. The injected units assume systemd as
PID 1. Alpine ≥3.20 ships systemd; older alpine + void + busybox-
init images won't honour the banger-* units.
- **First boot needs network access**. The first-boot sshd install
reaches out to the distro's package repo. VMs without NAT or
without the bridge reaching the internet time out. The marker file
stays in place so a later restart retries.
## Architecture
`internal/imagepull/` owns the pure mechanics:
`internal/imagepull/` owns the mechanics:
- **`Pull`** (`imagepull.go`) wraps `go-containerregistry`'s
`remote.Image` with the `linux/amd64` platform pinned. Layer
blobs are cached on disk via `cache.NewFilesystemCache` under
`<OCICacheDir>/blobs/` — Pull itself does not drain the layer
streams; that happens lazily during `Flatten`, and the cache
populates on read.
- **`Flatten`** (`flatten.go`) replays layers oldest-first into a
staging directory, applying whiteouts and rejecting unsafe paths.
Returns a `Metadata` map capturing per-file uid/gid/mode from
each tar header.
- **`BuildExt4`** (`ext4.go`) runs `mkfs.ext4 -F -d <staging>
-E root_owner=0:0` to populate the image file at create time —
no mount, no sudo, no loopback. Requires `e2fsprogs ≥ 1.43`
(`mkfs.ext4 -d` is the populate-at-create flag; nearly all
modern distros ship it).
- **`ApplyOwnership`** (`ownership.go`) streams a batched
`set_inode_field` script to `debugfs -w -f -` to rewrite per-file
uid/gid/mode to the captured tar-header values. Without this pass
the ext4 would carry the runner's on-disk uids.
- **`InjectGuestAgents`** (`inject.go`) uses the same `debugfs`
scripting to drop banger's guest-side assets into the pulled ext4
with root ownership:
- `/usr/local/bin/banger-vsock-agent`
- `/usr/local/libexec/banger-network-bootstrap`
- `/usr/local/libexec/banger-first-boot`
- `/etc/systemd/system/banger-{network,vsock-agent,first-boot}.service`
- enable-at-boot symlinks under `multi-user.target.wants/`
- `/etc/modules-load.d/banger-vsock.conf`
- `/var/lib/banger/first-boot-pending` (marker file)
- **`Pull`** wraps `go-containerregistry`'s `remote.Image` with the
`linux/amd64` platform pinned. Layer blobs cache under
`~/.cache/banger/oci/blobs/` and populate lazily during flatten.
- **`Flatten`** replays layers oldest-first into a staging directory,
applies whiteouts, rejects unsafe paths. Returns a `Metadata` map
of per-file uid/gid/mode from tar headers.
- **`BuildExt4`** runs `mkfs.ext4 -F -d <staging> -E root_owner=0:0`
at the size of the pre-truncated file — no mount, no sudo, no
loopback. Requires `e2fsprogs ≥ 1.43`.
- **`ApplyOwnership`** streams a batched `set_inode_field` script to
`debugfs -w` to rewrite per-file uid/gid/mode to the captured tar-
header values.
- **`InjectGuestAgents`** uses the same `debugfs` scripting to drop
banger's guest assets into the ext4 with root ownership:
vsock agent binary, network bootstrap + unit, first-boot script +
unit, `multi-user.target.wants` symlinks, vsock modules-load
config, `/var/lib/banger/first-boot-pending` marker.
`internal/daemon/images_pull.go` orchestrates:
`internal/daemon/images_pull.go` orchestrates `pullFromOCI`:
1. Parse + validate the OCI ref.
2. Derive a friendly default name (`debian-bookworm` for
`docker.io/library/debian:bookworm`) when `--name` is omitted.
3. Resolve kernel info via the shared `resolveKernelInputs` helper
(the same code path as `image register --kernel-ref`).
4. Stage at `<ImagesDir>/<id>.staging`; extract layers to a temp
tree under `os.TempDir` (bulk transient data stays off the
persistent state filesystem).
5. `imagepull.BuildExt4` produces `<staging>/rootfs.ext4`.
6. `ApplyOwnership` + `InjectGuestAgents` run in one finalize step.
7. `imagemgr.StageBootArtifacts` stages the kernel triple alongside.
8. Atomic `os.Rename(<staging>, <final>)` publishes the artifact dir.
9. Persist a `model.Image{Managed: true, …}` record.
Any failure removes the staging dir. Post-rename failures remove the
final dir and roll back the store write.
1. Parse + validate the OCI ref, derive a default name when `--name`
is omitted (`debian-bookworm` from
`docker.io/library/debian:bookworm`).
2. Resolve kernel info via `resolveKernelInputs` (auto-pulls from
`kernelcat` if `--kernel-ref` names a catalog entry that isn't
yet local).
3. Stage at `<ImagesDir>/<id>.staging`; extract layers to a temp
tree under `$TMPDIR`.
4. `BuildExt4``ApplyOwnership``InjectGuestAgents`.
5. `imagemgr.StageBootArtifacts` stages the kernel triple alongside.
6. Atomic `os.Rename` publishes the artifact dir.
7. Persist a `model.Image{Managed: true, …}` record.
## Guest-side boot sequence
On the first boot of a pulled image, systemd starts three banger
units in order:
On first boot of a pulled image:
1. **`banger-network.service`** — runs the bootstrap script that
parses `/etc/banger-network.conf` (written by banger's VM-create
lifecycle) and brings the guest interface up with the assigned IP.
2. **`banger-first-boot.service`** (only on first boot; removes its
own trigger file on success) — reads `/etc/os-release`, dispatches
to the native package manager, installs `openssh-server`, enables
`ssh.service` / `sshd.service`.
3. **`banger-vsock-agent.service`** — runs the health-check daemon
banger uses to confirm the VM is alive.
1. **`banger-network.service`** — brings the guest interface up with
the IP assigned by banger's VM-create lifecycle.
2. **`banger-first-boot.service`** (first boot only) — reads
`/etc/os-release`, dispatches to the native package manager,
installs `openssh-server`, enables `ssh.service`.
3. **`banger-vsock-agent.service`** — the health-check daemon banger
uses to confirm the VM is alive.
After first boot completes, subsequent boots skip the install step
entirely. Banger's host-side SSH polling (`guest.WaitForSSH`)
naturally retries until sshd is listening.
Subsequent boots skip step 2.
## Adding distro support
## Adding distro support to first-boot
`internal/imagepull/assets/first-boot.sh` is the POSIX-sh dispatch.
Add a new `ID=` branch and its install command to the `case` block,
then rebuild banger — the asset is `go:embed`-ed into the binary.
Add a new `ID=` branch and its install command, then rebuild banger
(the asset is `go:embed`-ed).
Supported `ID` values today: `debian`, `ubuntu`, `kali`, `raspbian`,
`linuxmint`, `pop`, `alpine`, `fedora`, `rhel`, `centos`, `rocky`,
`almalinux`, `arch`, `archlinux`, `manjaro`, `opensuse*`, `suse`.
Unknown distros fall back to `ID_LIKE`, then error clearly with a
pointer to edit the script.
Unknown distros fall back to `ID_LIKE`, then error cleanly.
## Paths
| What | Where | Purpose |
|------|-------|---------|
| Layer blob cache | `~/.cache/banger/oci/blobs/sha256/<hex>` | Re-pulls of the same image digest are local-only |
| Staging dir | `~/.local/state/banger/images/<id>.staging/` | Short-lived; atomic-renamed to `<id>/` on success |
| Staging rootfs tree | `$TMPDIR/banger-pull-<rand>/` | Extraction scratch space; removed after ext4 build |
| Published image | `~/.local/state/banger/images/<id>/rootfs.ext4` | Managed artifact stored alongside the kernel triple |
## Composition with `image build`
A pulled image boots as-is — ownership is correct, sshd installs on
first boot, banger's agents are in place. That means the existing
`image build --from-image` pipeline composes on top:
```bash
banger image build --from-image debian-bookworm --name debian-dev --docker
```
`image build` spins up a transient VM using the base image, runs
`scripts/customize.sh` over it, and saves the result as a new managed
image with the opinionated tooling (mise, opencode, claude, pi, tmux
plugins, optionally docker) layered on top.
| What | Where |
|------|-------|
| Layer blob cache | `~/.cache/banger/oci/blobs/sha256/<hex>` |
| Staging dir | `~/.local/state/banger/images/<id>.staging/` |
| Extraction scratch | `$TMPDIR/banger-pull-<rand>/` |
| Published image | `~/.local/state/banger/images/<id>/rootfs.ext4` |
## Tech debt
- **Auth**. When we add private-registry support, the natural path is
`authn.DefaultKeychain` from `go-containerregistry`, which already
honours `~/.docker/config.json` and the standard credential
helpers. No banger-specific config needed.
- **Cache eviction**. Layer blobs under `OCICacheDir` accumulate
forever. A `banger image cache prune` command is a cheap follow-up
when disk usage becomes a complaint.
- **First-boot timeout UX**. If you run `banger vm ssh` immediately
after `banger vm create`, the package install for `openssh-server`
may still be running and SSH will fail. Current mitigation: retry.
Better: a per-image `FirstBootPending` flag that tells the daemon
to extend its SSH wait timeout for the first boot, cleared on
success. Tracked but not implemented.
- **Non-systemd distros**. The guest agents assume systemd. Adding
- **Auth**. When we add private-registry support, the natural path
is `authn.DefaultKeychain`, which honours `~/.docker/config.json`
and the standard credential helpers.
- **Cache eviction**. OCI layer blobs accumulate forever. A `banger
image cache prune` command is a cheap follow-up when disk usage
becomes a complaint.
- **Non-systemd rootfses**. The guest agents assume systemd. Adding
openrc / s6 / busybox-init variants means keeping parallel unit
trees in `inject.go` keyed on `/etc/os-release`. Only pick up
when a user actually wants it.
trees keyed on `/etc/os-release`.
## Trust model
`image pull` delegates trust to the OCI registry the user selected.
`go-containerregistry` verifies layer digests against the manifest
during download, so a tampered mirror can't ship modified layers
without breaking the sha256 chain. Beyond that, banger does not
verify OCI image signatures (cosign/sigstore) — users who care should
verify their references out-of-band.
`image pull` (OCI path) delegates trust to the registry the user
selected. `go-containerregistry` verifies layer digests against the
manifest during download, so a tampered mirror can't ship modified
layers without breaking the sha256 chain. Banger does not verify OCI
image signatures (cosign/sigstore) — users who care should verify
references out-of-band.