thaloco/banger
1
Fork 0
Code Pull requests Releases Packages Activity Actions

docs: promote vm run + image catalog as the happy path

Browse source 
Lead the README with `banger vm run` (one command, auto-pull default
image + kernel from the catalogs), move `image register` / `image
build` / OCI-pull to a "power-user flows" section. Golden-image
content from customize.sh moves to the golden-image Dockerfile story.

New `docs/image-catalog.md` mirrors `docs/kernel-catalog.md` — the
bundle format, content-addressed filenames, publish flow, trust
model, R2 hosting. Cross-links with oci-import.md.

`docs/oci-import.md` refactored to document the OCI-pull path as the
fallthrough for arbitrary registry refs (it's the secondary path now
that the catalog covers the headline debian-bookworm case). Phase A
caveats removed — ownership fixup, agent injection, and first-boot
sshd install all landed.

AGENTS.md: promotes `vm run` as the smoke-test primitive, notes the
default-image auto-pull behaviour, and points at both catalog docs.

README shrinks 330 → 198 lines, mostly by removing the experimental
void/alpine sections (those flows still work as advanced scripts but
the README no longer advertises them).

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
This commit is contained in:
Thales Maciel 2026-04-18 15:33:30 -03:00
parent 75baf2e415
commit 8029b2e1bc
No known key found for this signature in database
GPG key ID: 33112E6833C34679
4 changed files with 404 additions and 450 deletions
Download patch file Download diff file Expand all files Collapse all files

21
AGENTS.md
View file

@ -7,7 +7,9 @@ Always run `make build` before commit.
- `cmd/banger` and `cmd/bangerd` are the main user entrypoints.
- `internal/` contains the daemon, CLI, RPC, storage, Firecracker integration, guest helpers, and the experimental web UI.
- `internal/daemon/` is the composition root; pure helpers live in its subpackages (`opstate`, `dmsnap`, `fcproc`, `imagemgr`, `session`, `workspace`). See `internal/daemon/ARCHITECTURE.md`.
- `scripts/` contains explicit manual helper workflows for rootfs and kernel preparation.
- `internal/imagecat/` and `internal/kernelcat/` embed the image + kernel catalogs.
- `images/golden/` is the Dockerfile for the `debian-bookworm` catalog entry.
- `scripts/` contains manual helper workflows for rootfs, kernel, and bundle preparation.
- `build/bin/` is the canonical source-checkout build output.
- `build/manual/` is the canonical source-checkout location for manual rootfs/kernel artifacts.
@ -17,19 +19,20 @@ Always run `make build` before commit.
- `make test` runs `go test ./...`.
- `make lint` runs `gofmt -l`, `go vet ./...`, and `shellcheck --severity=error` on `scripts/*.sh`. Run before commits.
- `./build/bin/banger doctor` checks host readiness.
- `./build/bin/banger image build --from-image <image>` builds a managed image from an existing registered image.
- `./build/bin/banger vm run` is the primary user-facing entry point — auto-pulls the default image + kernel from the catalogs if missing.
- `./build/bin/banger image pull <name>` uses the bundle catalog (fast) when `<name>` is a catalog entry, or falls through to the OCI path for arbitrary registry refs. See `docs/image-catalog.md` and `docs/oci-import.md`.
- `./build/bin/banger image register ...` registers an unmanaged host-side image stack.
- `./build/bin/banger image build --from-image <image>` builds a managed image from an existing one.
- `./build/bin/banger image promote <image>` copies an unmanaged image into daemon-owned managed artifacts.
- `scripts/make-generic-kernel.sh` builds a Firecracker-optimized vmlinux from upstream sources (no initrd, all drivers built-in). This is the recommended kernel for OCI-pulled images.
- `make void-kernel`, `make rootfs-void`, and `make void-register` drive the experimental Void flow under `./build/manual`.
- `scripts/publish-kernel.sh <name>` packages a locally-imported kernel and uploads it to the catalog; see `docs/kernel-catalog.md`.
- `banger image pull <oci-ref> --kernel-ref <name>` pulls a rootfs from any OCI registry; see `docs/oci-import.md` (experimental — file-ownership caveat).
- `scripts/make-generic-kernel.sh` builds a Firecracker-optimized vmlinux from upstream sources. `scripts/publish-kernel.sh <name>` publishes it to the kernel catalog.
- `scripts/publish-golden-image.sh` rebuilds + publishes the golden image bundle and patches the image catalog.
## Image Model
- Managed images own the full boot set: rootfs, optional work-seed, kernel, optional initrd, and optional modules.
- There is no runtime bundle and no auto-registered default image from disk paths.
- `default_image_name` selects a registered image only.
- The image catalog ships pre-built bundles. `vm run` auto-pulls the default catalog entry; `image pull <name>` can be invoked explicitly.
- `default_image_name` defaults to `debian-bookworm`. On miss, the daemon auto-pulls from `imagecat` before surfacing "not found".
- Kernel references follow the same auto-pull pattern against `kernelcat`.
## Config
@ -50,7 +53,7 @@ Always run `make build` before commit.
## Testing Guidance
- Primary automated coverage is `go test ./...`.
- For lifecycle changes, smoke-test with `vm create`, `vm ssh`, `vm stop`, and `vm delete`.
- For lifecycle changes, smoke-test with `vm run` end-to-end (covers create + start + boot + ssh).
- If guest provisioning changes, document whether existing images must be rebuilt or recreated.
## Security

448
README.md
View file

@ -1,330 +1,198 @@
# banger
`banger` manages Firecracker development VMs with a local daemon, managed image artifacts, and an experimental localhost web UI.
One-command development sandboxes on Firecracker microVMs.
## Quick start
```bash
make install
banger vm run --name sandbox
```
`banger vm run` auto-pulls the default golden image (Debian bookworm
with systemd, sshd, Docker CE, git, jq, mise, and the usual dev tools)
and kernel from the embedded catalog if they aren't already local,
creates a VM, starts it, and drops you into an interactive ssh
session. First run takes a couple minutes (bundle download);
subsequent `vm run`s are seconds.
## Requirements
- Linux with `/dev/kvm`
- `sudo`
- Firecracker installed on `PATH`, or `firecracker_bin` set in config
- The usual host tools checked by `./build/bin/banger doctor`
- Firecracker on `PATH`, or `firecracker_bin` set in config
- host tools checked by `banger doctor`
`banger` now owns complete managed image sets. A managed image includes:
- `rootfs`
- optional `work-seed`
- `kernel`
- optional `initrd`
- optional `modules`
There is no runtime bundle anymore.
## Build
```bash
make build
```
This writes:
- `./build/bin/banger`
- `./build/bin/bangerd`
- `./build/bin/banger-vsock-agent`
## Install
## Build + install
```bash
make install
```
That installs:
Installs:
- `banger`
- `bangerd`
- the `banger-vsock-agent` companion helper under `../lib/banger/`
- `banger` (CLI)
- `bangerd` (daemon, auto-starts on first CLI call)
- `banger-vsock-agent` (companion, under `$PREFIX/lib/banger/`)
## `vm run`
One command, three modes:
```bash
banger vm run # bare sandbox — drops into ssh
banger vm run ./repo # workspace at /root/repo — drops into ssh
banger vm run ./repo -- make test # workspace + run command, exit with its status
```
- Bare mode gives you a clean shell.
- Workspace mode (with a path) copies the repo's tracked + untracked
non-ignored files into `/root/repo` and kicks off a best-effort
mise tooling bootstrap from the repo's `.mise.toml` /
`.tool-versions`. Log: `/root/.cache/banger/vm-run-tooling-<repo>.log`.
- Command mode (`-- <cmd>`) runs the command in the guest; exit code
propagates through `banger`.
Disconnecting from an interactive session leaves the VM running. Use
`vm stop` / `vm delete` to clean up.
`--branch` and `--from` apply only to workspace mode.
## Image catalog
`banger image pull <name>` resolves `<name>` in the embedded catalog
and fetches a pre-built bundle (rootfs.ext4 + manifest, tar+zstd). The
kernel referenced by the manifest auto-pulls too. `vm run` calls this
for you on demand.
Today's catalog:
| Name | Distro | Kernel |
|------|--------|--------|
| `debian-bookworm` | Debian 12 slim + sshd + docker + dev tools | `generic-6.12` |
The catalog ships embedded in the banger binary. See
[`docs/image-catalog.md`](docs/image-catalog.md) for maintenance.
## Power-user flows
Skip this section if `vm run` is enough.
### `vm create` — low-level primitive
For scripting or `--no-start` provisioning:
```bash
banger vm create --image debian-bookworm --name testbox --no-start
banger vm start testbox
banger vm ssh testbox
banger vm stop testbox
```
### `image pull <oci-ref>` — arbitrary container images
For images outside the catalog, pull from any OCI registry:
```bash
banger image pull docker.io/library/alpine:3.20 --kernel-ref generic-6.12
```
Layers are flattened, ownership is fixed, banger's guest agents are
injected, and a first-boot service installs `openssh-server` via the
guest's package manager. See [`docs/oci-import.md`](docs/oci-import.md)
for supported distros and caveats.
### `image register` — existing host-side stack
```bash
banger image register --name base \
--rootfs /abs/path/rootfs.ext4 \
--kernel-ref generic-6.12
```
### `image build --from-image` — derived images
```bash
banger image build --name devbox --from-image debian-bookworm --docker
```
Spins up a transient VM from a base image, applies opinionated
customisation (mise, claude, pi, tmux plugins), saves a new managed
image.
### Workspace + session primitives
Long-lived guest commands managed by the daemon, attachable over a
local Unix socket bridge:
```bash
banger vm workspace prepare <vm> ./other-repo --guest-path /root/repo
banger vm session start <vm> --name planner --cwd /root/repo --stdin-mode pipe -- pi --mode rpc
banger vm session attach <vm> planner
banger vm session logs <vm> planner --stream stderr
banger vm session stop <vm> planner
```
For ACP-aware host tooling: `banger vm acp <vm>` bridges stdio to
guest `opencode acp` over SSH.
## Config
Config lives at `~/.config/banger/config.toml`.
Config lives at `~/.config/banger/config.toml`. All keys optional.
Supported keys:
Commonly set:
- `log_level`
- `web_listen_addr`
- `firecracker_bin`
- `ssh_key_path`
- `default_image_name`
- `auto_stop_stale_after`
- `stats_poll_interval`
- `metrics_poll_interval`
- `bridge_name`
- `bridge_ip`
- `cidr`
- `tap_pool_size`
- `default_dns`
- `default_image_name` — image to use when `--image` is omitted
(defaults to `debian-bookworm`, auto-pulled from the catalog if not
local).
- `ssh_key_path` — host SSH key; if unset banger creates
`~/.config/banger/ssh/id_ed25519`.
- `firecracker_bin` — override the auto-resolved `PATH` lookup.
- `web_listen_addr` — experimental web UI (default
`127.0.0.1:7777`; set to `""` to disable).
- Network: `bridge_name`, `bridge_ip`, `cidr`, `tap_pool_size`,
`default_dns`.
If `ssh_key_path` is unset, banger creates and uses:
Full key list in `internal/config/config.go`.
- `~/.config/banger/ssh/id_ed25519`
## Credential sync
`default_image_name` now only means “use this registered image when `vm create` omits `--image`”. The daemon does not auto-register images from host paths.
If these host auth files exist, banger syncs them into the guest at
VM start:
## Core Workflow
| Host | Guest |
|------|-------|
| `~/.local/share/opencode/auth.json` | `/root/.local/share/opencode/auth.json` |
| `~/.claude/.credentials.json` | `/root/.claude/.credentials.json` |
| `~/.pi/agent/auth.json` | `/root/.pi/agent/auth.json` |
Check the host:
```bash
./build/bin/banger doctor
```
Register an existing host-side image stack:
```bash
./build/bin/banger image register \
--name base \
--rootfs /abs/path/rootfs.ext4 \
--kernel /abs/path/vmlinux \
--initrd /abs/path/initrd.img \
--modules /abs/path/modules
```
Or pull a pre-built kernel from the catalog and reference it by name:
```bash
./build/bin/banger kernel list --available
./build/bin/banger kernel pull generic-6.12
./build/bin/banger image register \
--name base \
--rootfs /abs/path/rootfs.ext4 \
--kernel-ref generic-6.12
```
See [`docs/kernel-catalog.md`](docs/kernel-catalog.md) for catalog
maintenance.
Or pull a rootfs directly from any OCI registry (Docker Hub, GHCR, …):
```bash
./build/bin/banger image pull docker.io/library/debian:bookworm \
--kernel-ref generic-6.12
```
`image pull` downloads the image, flattens its layers into an ext4
rootfs, applies tar-header ownership via debugfs, and pre-injects
banger's guest agents (vsock agent + network bootstrap + a first-boot
unit that installs `openssh-server` via the guest's native package
manager). Boots as a banger VM directly, no `image build` step
required. See [`docs/oci-import.md`](docs/oci-import.md) for
supported distros and current limitations.
Build a managed image from an existing registered image:
```bash
./build/bin/banger image build \
--name devbox \
--from-image base \
--docker
```
Promote an unmanaged image into daemon-owned managed artifacts:
```bash
./build/bin/banger image promote base
```
Spin up a sandbox VM and drop straight into it:
```bash
./build/bin/banger vm run # bare sandbox, interactive ssh
./build/bin/banger vm run ../some-repo # workspace at /root/repo, interactive ssh
./build/bin/banger vm run ../some-repo -- make test # workspace, run command, exit with its status
```
`vm run` creates a VM, prepares a workspace if you pass a path, and then either drops you into an interactive ssh session or runs the `--`-delimited command to completion. The command's exit code propagates through `banger`. Disconnecting from the interactive session leaves the VM running; use `vm stop` / `vm delete` to clean up.
When you pass a path, `vm run` copies a git checkout plus tracked and untracked non-ignored files into `/root/repo`, then kicks off a best-effort `mise` tooling bootstrap that runs asynchronously inside the guest (log at `/root/.cache/banger/vm-run-tooling-<repo>.log`). The bootstrap is skipped in bare and command modes. Flags like `--branch` and `--from` require a path.
For scripting or lower-level control, `vm create` remains available as a primitive (use `--no-start` when you just want to provision):
```bash
./build/bin/banger vm create --image devbox --name testbox --no-start
./build/bin/banger vm start testbox
./build/bin/banger vm ssh testbox
./build/bin/banger vm stop testbox
```
`vm create` stays synchronous by default, but on a TTY it now shows live progress until the VM is fully ready.
For ACP-aware host tools, `./build/bin/banger vm acp <vm-name>` bridges stdio to guest `opencode acp` over SSH. It uses `/root/repo` when that checkout exists, otherwise `/root`, and `--cwd` lets you override the guest working directory explicitly.
If you want reusable orchestration primitives instead of the `vm run` convenience flow, use the daemon-backed workspace and session commands directly:
```bash
./build/bin/banger vm workspace prepare <vm-name>
./build/bin/banger vm workspace prepare <vm-name> ../other-repo --guest-path /root/repo --readonly
./build/bin/banger vm session start <vm-name> --name planner --cwd /root/repo --stdin-mode pipe -- pi --mode rpc --no-session
./build/bin/banger vm session list <vm-name>
./build/bin/banger vm session attach <vm-name> planner
./build/bin/banger vm session logs <vm-name> planner --stream stderr
./build/bin/banger vm session stop <vm-name> planner
```
`vm workspace prepare` materializes a local git checkout into a running VM. The default guest path is `/root/repo` and the default mode is a shallow metadata copy plus tracked and untracked non-ignored overlay. Repositories with git submodules must use `--mode full_copy`; the metadata-based modes still reject them.
`vm session start` creates a daemon-managed long-lived guest command. The daemon preflights that the requested guest `cwd` exists and that the main command, plus any repeated `--require-command` entries, exist in guest `PATH` before launch. Use `--stdin-mode pipe` when you need live `attach`; otherwise use the default detached mode and inspect sessions with `list`, `show`, `logs`, `stop`, and `kill`.
`vm session attach` is currently exclusive and same-host only. The daemon exposes a local Unix socket bridge using `stdio_mux_v1`, so only one active attach is allowed at a time. Pipe-mode sessions keep enough guest-side state for the daemon to rebuild that bridge after a daemon restart.
Host-side changes take effect after the VM restarts. Session/history
directories are not copied.
## Web UI (experimental)
`bangerd` serves an experimental local web UI by default at:
- `http://127.0.0.1:7777`
The UI is convenient for local observability but is **not a stable or
supported interface**. Its endpoints, layout, and behaviour may change
without notice, and it has not been hardened for anything beyond single-user
localhost use. Do not expose the listen address to a shared network.
See the effective URL with:
```bash
./build/bin/banger daemon status
```
Disable it with:
```toml
web_listen_addr = ""
```
## Guest Services
Provisioned glibc-backed images include:
- `banger-vsock-agent`
- guest networking bootstrap
- `mise`
- `opencode`
- `claude`
- `pi`
- a default guest `opencode` service on `0.0.0.0:4096`
Alpine currently remains `opencode`-only.
If these host auth files exist, `banger` syncs them into the guest on VM start:
- `~/.local/share/opencode/auth.json` -> `/root/.local/share/opencode/auth.json`
- `~/.claude/.credentials.json` -> `/root/.claude/.credentials.json`
- `~/.pi/agent/auth.json` -> `/root/.pi/agent/auth.json`
Changes on the host take effect after the VM is restarted. Session/history directories are not copied.
From the host:
```bash
./build/bin/banger vm ports testbox
opencode attach http://<guest-ip>:4096
```
## Manual Helpers
The shell helpers are now explicit manual workflows under `./build/manual`.
Rebuild a Debian-style manual rootfs:
```bash
make rootfs ARGS='--base-rootfs /abs/path/rootfs.ext4 --kernel /abs/path/vmlinux --initrd /abs/path/initrd.img --modules /abs/path/modules'
```
The output lands in:
- `./build/manual/rootfs-docker.ext4`
- `./build/manual/rootfs-docker.work-seed.ext4`
## Experimental Void Flow
Stage a Void kernel:
```bash
make void-kernel
```
Build the experimental Void rootfs:
```bash
make rootfs-void
```
Register it:
```bash
make void-register
```
That flow uses:
- `./build/manual/void-kernel/`
- `./build/manual/rootfs-void.ext4`
- `./build/manual/rootfs-void.work-seed.ext4`
## Experimental Alpine Flow
Stage an Alpine virt kernel:
```bash
make alpine-kernel
```
Build the experimental Alpine rootfs:
```bash
make rootfs-alpine
```
Register it:
```bash
make alpine-register
```
Create a VM from it:
```bash
./build/bin/banger vm create --image alpine --name alpine-dev
```
That flow uses:
- `./build/manual/alpine-kernel/`
- `./build/manual/rootfs-alpine.ext4`
- `./build/manual/rootfs-alpine.work-seed.ext4`
The experimental Alpine flow stages a pinned Alpine release by default. Override
that pin with `ALPINE_RELEASE=...` when running the `make alpine-kernel` and
`make rootfs-alpine` helpers if you need a different patch release.
Alpine support currently applies to the explicit register-and-run flow above.
The generic `banger image build --from-image ...` path remains Debian/systemd-
oriented and should not be treated as an Alpine image builder.
`bangerd` serves a local web UI at `http://127.0.0.1:7777` by default.
Convenient for local observability, **not a stable interface**. Do
not expose the listen address to a shared network.
## Security
Guest VMs are single-user development sandboxes, not multi-tenant servers.
Every provisioned image is configured with:
Guest VMs are single-user development sandboxes, not multi-tenant
servers. Every provisioned image is configured with:
```
PermitRootLogin yes
StrictModes no
```
This is intentional. The host SSH key is the only authentication mechanism,
no password auth is enabled, and VMs are reachable only through the host
bridge network (`172.16.0.0/24` by default). Do not expose the bridge
interface or the VM guest IPs to an untrusted network.
The host SSH key is the only authentication mechanism, no password
auth is enabled, and VMs are reachable only through the host bridge
network (`172.16.0.0/24` by default). Do not expose the bridge
interface or guest IPs to an untrusted network.
## Notes
- Firecracker is resolved from `PATH` by default.
- Managed image delete removes the daemon-owned artifact dir.
- The companion vsock helper is internal to the install/build layout, not a user-configured runtime path.
- Layer blob cache for OCI pulls lives under `~/.cache/banger/oci/`.
- Image bundle cache doesn't exist — bundles are extracted directly
into the image store; re-pulls download fresh.

123
docs/image-catalog.md Normal file
View file

@ -0,0 +1,123 @@
# Image catalog
The image catalog ships pre-built banger rootfs bundles so users don't
have to register or build anything. It's the fast path behind
`banger vm run` (auto-pull) and `banger image pull <name>`. The
catalog is embedded into the banger binary and updated each release.
End-user flow:
```bash
banger image pull debian-bookworm # explicit
banger vm run --name sandbox # implicit (auto-pulls)
```
## Architecture
Two parts — the same shape as the kernel catalog:
1. **`internal/imagecat/catalog.json`** — JSON manifest embedded into
the banger binary via `go:embed`. Each entry: name, distro, arch,
kernel_ref (a `kernelcat` entry name), tarball URL, tarball
sha256, size.
2. **Tarballs at `https://images.thaloco.com/`** — Cloudflare R2
bucket `banger-images`, fronted by a public custom domain. Each
tarball is `<name>-<arch>-<sha256-prefix>.tar.zst` (content-
addressed filename so CDN edge cache can never serve stale bytes
for the URL the catalog points at). Contents at the archive root:
`rootfs.ext4` (finalized: flattened + ownership-fixed + agent-
injected at build time) and `manifest.json`.
The `banger image pull` bundle path streams the tarball, verifies
sha256 against the catalog entry, extracts both files into a staging
dir, resolves the kernel via `kernel_ref` (auto-pulling from
`kernelcat` if the user hasn't pulled it yet), stages boot artifacts
alongside, and registers the result as a managed image.
The same `image pull` command transparently falls through to the
existing OCI-pull path when `<name>` doesn't match a catalog entry —
see [`docs/oci-import.md`](oci-import.md).
## Adding or updating an entry
The repo has no CI for bundle publishing yet. Catalog updates are
manual.
```bash
# 1. Build the bundle + upload + patch catalog.json in one shot.
scripts/publish-golden-image.sh
# 2. Review and commit the catalog change.
git diff -- internal/imagecat/catalog.json
git add internal/imagecat/catalog.json
git commit -m 'imagecat: publish debian-bookworm'
# 3. Rebuild so the new catalog is embedded.
make build
```
`scripts/publish-golden-image.sh` wraps `scripts/make-golden-bundle.sh`
(which runs `docker build` on `images/golden/Dockerfile` then pipes
`docker export` into `banger internal make-bundle`), computes the
bundle's sha256, uses the first 12 hex chars as a cache-busting
filename suffix, uploads via `rclone` to R2, HEAD-checks the public
URL, and patches `internal/imagecat/catalog.json`.
Environment overrides if the defaults need to change:
`RCLONE_REMOTE`, `RCLONE_BUCKET`, `BASE_URL`.
`--skip-upload` builds the bundle into `dist/` and stops — useful for
local testing without touching R2 or the catalog.
## Bundle format
A bundle is a tar+zstd archive with exactly two entries at the root:
```
rootfs.ext4 # finalized banger rootfs
manifest.json # {name, distro, arch, kernel_ref, description}
```
`rootfs.ext4` is fully prepared at build time: ownership fixed via
`debugfs sif`, banger guest agents (vsock agent, network bootstrap,
first-boot unit) already injected and enabled in
`multi-user.target.wants`. The pull path only has to place the file
and register the image — no mkfs, no ownership pass, no injection on
the daemon host.
## Removing an entry
1. Remove the entry from `internal/imagecat/catalog.json` and commit.
2. Delete the tarball from R2:
`rclone delete banger-images:banger-images/<name>-<arch>-<hash>.tar.zst`.
3. Rebuild banger.
Already-pulled local images are not invalidated — users keep using
them until they run `banger image delete <name>`.
## Versioning conventions
- **Entry names**: `<distro>-<release>` (e.g. `debian-bookworm`).
Per-release names make it trivial to publish `debian-trixie`
alongside without collisions.
- **Content-addressed filenames**: the `-<sha256-prefix>` suffix is
mandatory (set by `publish-golden-image.sh`). Never reuse a URL for
different bytes.
- **Architecture**: `x86_64` only today. The `arch` field is additive
— adding `arm64` is a config change, not a schema change.
## Trust model
Same as the kernel catalog: the embedded `catalog.json` carries each
bundle's sha256, and `imagecat.Fetch` rejects any download whose hash
doesn't match. This protects against transport corruption and against
an attacker swapping an R2 object without landing a commit in the
banger repo. GPG/sigstore signing is deferred until banger is public
and the threat model justifies the operational overhead.
## Hosting
Tarballs live in Cloudflare R2 (bucket `banger-images`), served at
`images.thaloco.com`. The bucket is publicly readable; writes require
the R2 API token configured on the `banger-images` rclone remote.

262
docs/oci-import.md
View file

@ -1,193 +1,153 @@
# OCI import (`banger image pull`)
`banger image pull <oci-ref>` downloads a container image from any
OCI-compatible registry (Docker Hub, GHCR, quay.io, self-hosted, …),
flattens its layers into an ext4 rootfs, and registers the result as
a managed banger image.
`banger image pull` has two paths. The primary one — catalog bundle —
is documented in [`docs/image-catalog.md`](image-catalog.md). This
doc covers the fallthrough: OCI-registry pull for arbitrary container
images.
Paired with the kernel catalog, this dissolves the "where do I get a
rootfs" bottleneck for most users — any distro that ships an official
container image can now boot (eventually) as a banger VM.
## When to use it
Use the OCI path when you need a distro or image that isn't in the
catalog. The catalog covers the common happy path
(`debian-bookworm`); anything else (`alpine`, `fedora`, `ubuntu`,
custom corporate images) goes through OCI pull.
```bash
banger kernel pull void-6.12
banger image pull docker.io/library/debian:bookworm --kernel-ref void-6.12
banger image list # debian-bookworm appears, Managed=true
banger image pull docker.io/library/alpine:3.20 --kernel-ref generic-6.12
banger image pull ghcr.io/myorg/devimg:v2 --kernel-ref generic-6.12
```
`banger image pull` dispatches based on the reference:
- `banger image pull debian-bookworm` → catalog (fast path).
- `banger image pull docker.io/library/foo:bar` → OCI (anything not
in the catalog).
## What works
- Pulling any public OCI image that exposes a `linux/amd64` manifest.
- Any public OCI image that exposes a `linux/amd64` manifest.
- Correct layer replay with whiteout semantics (`.wh.*` deletes,
`.wh..wh..opq` opaque-dir markers).
- Path-traversal and relative-symlink-escape protection.
- Content-aware default sizing (`content × 1.25`, floor 1 GiB).
- Layer caching on disk, keyed by blob SHA256.
- **File ownership preservation.** Tar-header uid/gid/mode is captured
during flatten and applied to the resulting ext4 via a `debugfs`
pass, so setuid binaries (`sudo`, `passwd`) and root-owned config
files (`/etc/shadow`, `/etc/sudoers`) end up correctly owned.
- **Banger guest agents pre-injected.** The pulled ext4 ships with
`/usr/local/bin/banger-vsock-agent`, `banger-network.service`, and
`banger-vsock-agent.service` already in place and enabled.
- **First-boot sshd install.** A one-shot systemd service installs
`openssh-server` via the guest's package manager on first boot —
apt-get / apk / dnf / pacman / zypper dispatch based on
`/etc/os-release`. Subsequent boots skip the install.
- Piping pulled images into the existing `banger image build
--from-image` flow.
- Content-aware default sizing (`content × 1.5`, floor 1 GiB).
- Layer caching on disk, keyed by blob sha256.
- **Ownership preservation** — tar-header uid/gid/mode captured
during flatten, applied to the ext4 via a `debugfs` pass, so
setuid binaries (`sudo`, `passwd`) and root-owned config
(`/etc/shadow`, `/etc/sudoers`) end up correctly owned.
- **Pre-injected banger agents** — the pulled ext4 ships with
`banger-vsock-agent`, `banger-network.service`, and the
`banger-first-boot` unit already enabled.
- **First-boot sshd install** — a one-shot systemd service installs
`openssh-server` via the guest's package manager on first boot.
Dispatches on `/etc/os-release` → `apt-get` / `apk` / `dnf` /
`pacman` / `zypper`. Subsequent boots skip the install.
- Composition with `image build --from-image`.
## What doesn't yet work
- **Private registries**. Auth is not implemented; anonymous pulls
only. Docker Hub, GHCR (public), quay.io (public), etc. all work.
- **Non-`linux/amd64` platforms**. The kernel catalog is x86_64-only,
so pulled rootfses match. `arm64` is additive in the schema; wire-
up lands when a user needs it.
- **Non-systemd distros.** The injected units assume systemd as PID 1.
Alpine ≥3.20 ships systemd; older alpine + void + busybox-init
images won't honour the banger-network / banger-first-boot units.
- **First boot needs network access.** The provisioning step reaches
out to the distro's package repo to install openssh-server. VMs
without NAT or without the bridge reaching the internet will time
out on first boot. The marker file stays in place so a later boot
retries.
- **Private registries**. Anonymous pulls only. Docker Hub, GHCR
(public), quay.io (public) all work. Adding auth via
`authn.DefaultKeychain` (from `go-containerregistry`) is a cheap
follow-up when someone needs it.
- **Non-`linux/amd64`**. The kernel catalog is x86_64-only, so pulled
rootfses match. `arm64` is additive in the schema.
- **Non-systemd rootfses**. The injected units assume systemd as
PID 1. Alpine ≥3.20 ships systemd; older alpine + void + busybox-
init images won't honour the banger-* units.
- **First boot needs network access**. The first-boot sshd install
reaches out to the distro's package repo. VMs without NAT or
without the bridge reaching the internet time out. The marker file
stays in place so a later restart retries.
## Architecture
`internal/imagepull/` owns the pure mechanics:
`internal/imagepull/` owns the mechanics:
- **`Pull`** (`imagepull.go`) wraps `go-containerregistry`'s
`remote.Image` with the `linux/amd64` platform pinned. Layer
blobs are cached on disk via `cache.NewFilesystemCache` under
`<OCICacheDir>/blobs/` — Pull itself does not drain the layer
streams; that happens lazily during `Flatten`, and the cache
populates on read.
- **`Flatten`** (`flatten.go`) replays layers oldest-first into a
staging directory, applying whiteouts and rejecting unsafe paths.
Returns a `Metadata` map capturing per-file uid/gid/mode from
each tar header.
- **`BuildExt4`** (`ext4.go`) runs `mkfs.ext4 -F -d <staging>
-E root_owner=0:0` to populate the image file at create time —
no mount, no sudo, no loopback. Requires `e2fsprogs ≥ 1.43`
(`mkfs.ext4 -d` is the populate-at-create flag; nearly all
modern distros ship it).
- **`ApplyOwnership`** (`ownership.go`) streams a batched
`set_inode_field` script to `debugfs -w -f -` to rewrite per-file
uid/gid/mode to the captured tar-header values. Without this pass
the ext4 would carry the runner's on-disk uids.
- **`InjectGuestAgents`** (`inject.go`) uses the same `debugfs`
scripting to drop banger's guest-side assets into the pulled ext4
with root ownership:
- `/usr/local/bin/banger-vsock-agent`
- `/usr/local/libexec/banger-network-bootstrap`
- `/usr/local/libexec/banger-first-boot`
- `/etc/systemd/system/banger-{network,vsock-agent,first-boot}.service`
- enable-at-boot symlinks under `multi-user.target.wants/`
- `/etc/modules-load.d/banger-vsock.conf`
- `/var/lib/banger/first-boot-pending` (marker file)
- **`Pull`** wraps `go-containerregistry`'s `remote.Image` with the
`linux/amd64` platform pinned. Layer blobs cache under
`~/.cache/banger/oci/blobs/` and populate lazily during flatten.
- **`Flatten`** replays layers oldest-first into a staging directory,
applies whiteouts, rejects unsafe paths. Returns a `Metadata` map
of per-file uid/gid/mode from tar headers.
- **`BuildExt4`** runs `mkfs.ext4 -F -d <staging> -E root_owner=0:0`
at the size of the pre-truncated file — no mount, no sudo, no
loopback. Requires `e2fsprogs ≥ 1.43`.
- **`ApplyOwnership`** streams a batched `set_inode_field` script to
`debugfs -w` to rewrite per-file uid/gid/mode to the captured tar-
header values.
- **`InjectGuestAgents`** uses the same `debugfs` scripting to drop
banger's guest assets into the ext4 with root ownership:
vsock agent binary, network bootstrap + unit, first-boot script +
unit, `multi-user.target.wants` symlinks, vsock modules-load
config, `/var/lib/banger/first-boot-pending` marker.
`internal/daemon/images_pull.go` orchestrates:
`internal/daemon/images_pull.go` orchestrates `pullFromOCI`:
1. Parse + validate the OCI ref.
2. Derive a friendly default name (`debian-bookworm` for
`docker.io/library/debian:bookworm`) when `--name` is omitted.
3. Resolve kernel info via the shared `resolveKernelInputs` helper
(the same code path as `image register --kernel-ref`).
4. Stage at `<ImagesDir>/<id>.staging`; extract layers to a temp
tree under `os.TempDir` (bulk transient data stays off the
persistent state filesystem).
5. `imagepull.BuildExt4` produces `<staging>/rootfs.ext4`.
6. `ApplyOwnership` + `InjectGuestAgents` run in one finalize step.
7. `imagemgr.StageBootArtifacts` stages the kernel triple alongside.
8. Atomic `os.Rename(<staging>, <final>)` publishes the artifact dir.
9. Persist a `model.Image{Managed: true, …}` record.
Any failure removes the staging dir. Post-rename failures remove the
final dir and roll back the store write.
1. Parse + validate the OCI ref, derive a default name when `--name`
is omitted (`debian-bookworm` from
`docker.io/library/debian:bookworm`).
2. Resolve kernel info via `resolveKernelInputs` (auto-pulls from
`kernelcat` if `--kernel-ref` names a catalog entry that isn't
yet local).
3. Stage at `<ImagesDir>/<id>.staging`; extract layers to a temp
tree under `$TMPDIR`.
4. `BuildExt4` → `ApplyOwnership` → `InjectGuestAgents`.
5. `imagemgr.StageBootArtifacts` stages the kernel triple alongside.
6. Atomic `os.Rename` publishes the artifact dir.
7. Persist a `model.Image{Managed: true, …}` record.
## Guest-side boot sequence
On the first boot of a pulled image, systemd starts three banger
units in order:
On first boot of a pulled image:
1. **`banger-network.service`** — runs the bootstrap script that
parses `/etc/banger-network.conf` (written by banger's VM-create
lifecycle) and brings the guest interface up with the assigned IP.
2. **`banger-first-boot.service`** (only on first boot; removes its
own trigger file on success) — reads `/etc/os-release`, dispatches
to the native package manager, installs `openssh-server`, enables
`ssh.service` / `sshd.service`.
3. **`banger-vsock-agent.service`** — runs the health-check daemon
banger uses to confirm the VM is alive.
1. **`banger-network.service`** — brings the guest interface up with
the IP assigned by banger's VM-create lifecycle.
2. **`banger-first-boot.service`** (first boot only) — reads
`/etc/os-release`, dispatches to the native package manager,
installs `openssh-server`, enables `ssh.service`.
3. **`banger-vsock-agent.service`** — the health-check daemon banger
uses to confirm the VM is alive.
After first boot completes, subsequent boots skip the install step
entirely. Banger's host-side SSH polling (`guest.WaitForSSH`)
naturally retries until sshd is listening.
Subsequent boots skip step 2.
## Adding distro support
## Adding distro support to first-boot
`internal/imagepull/assets/first-boot.sh` is the POSIX-sh dispatch.
Add a new `ID=` branch and its install command to the `case` block,
then rebuild banger — the asset is `go:embed`-ed into the binary.
Add a new `ID=` branch and its install command, then rebuild banger
(the asset is `go:embed`-ed).
Supported `ID` values today: `debian`, `ubuntu`, `kali`, `raspbian`,
`linuxmint`, `pop`, `alpine`, `fedora`, `rhel`, `centos`, `rocky`,
`almalinux`, `arch`, `archlinux`, `manjaro`, `opensuse*`, `suse`.
Unknown distros fall back to `ID_LIKE`, then error clearly with a
pointer to edit the script.
Unknown distros fall back to `ID_LIKE`, then error cleanly.
## Paths
| What | Where | Purpose |
|------|-------|---------|
| Layer blob cache | `~/.cache/banger/oci/blobs/sha256/<hex>` | Re-pulls of the same image digest are local-only |
| Staging dir | `~/.local/state/banger/images/<id>.staging/` | Short-lived; atomic-renamed to `<id>/` on success |
| Staging rootfs tree | `$TMPDIR/banger-pull-<rand>/` | Extraction scratch space; removed after ext4 build |
| Published image | `~/.local/state/banger/images/<id>/rootfs.ext4` | Managed artifact stored alongside the kernel triple |
## Composition with `image build`
A pulled image boots as-is — ownership is correct, sshd installs on
first boot, banger's agents are in place. That means the existing
`image build --from-image` pipeline composes on top:
```bash
banger image build --from-image debian-bookworm --name debian-dev --docker
```
`image build` spins up a transient VM using the base image, runs
`scripts/customize.sh` over it, and saves the result as a new managed
image with the opinionated tooling (mise, opencode, claude, pi, tmux
plugins, optionally docker) layered on top.
| What | Where |
|------|-------|
| Layer blob cache | `~/.cache/banger/oci/blobs/sha256/<hex>` |
| Staging dir | `~/.local/state/banger/images/<id>.staging/` |
| Extraction scratch | `$TMPDIR/banger-pull-<rand>/` |
| Published image | `~/.local/state/banger/images/<id>/rootfs.ext4` |
## Tech debt
- **Auth**. When we add private-registry support, the natural path is
`authn.DefaultKeychain` from `go-containerregistry`, which already
honours `~/.docker/config.json` and the standard credential
helpers. No banger-specific config needed.
- **Cache eviction**. Layer blobs under `OCICacheDir` accumulate
forever. A `banger image cache prune` command is a cheap follow-up
when disk usage becomes a complaint.
- **First-boot timeout UX**. If you run `banger vm ssh` immediately
after `banger vm create`, the package install for `openssh-server`
may still be running and SSH will fail. Current mitigation: retry.
Better: a per-image `FirstBootPending` flag that tells the daemon
to extend its SSH wait timeout for the first boot, cleared on
success. Tracked but not implemented.
- **Non-systemd distros**. The guest agents assume systemd. Adding
- **Auth**. When we add private-registry support, the natural path
is `authn.DefaultKeychain`, which honours `~/.docker/config.json`
and the standard credential helpers.
- **Cache eviction**. OCI layer blobs accumulate forever. A `banger
image cache prune` command is a cheap follow-up when disk usage
becomes a complaint.
- **Non-systemd rootfses**. The guest agents assume systemd. Adding
openrc / s6 / busybox-init variants means keeping parallel unit
trees in `inject.go` keyed on `/etc/os-release`. Only pick up
when a user actually wants it.
trees keyed on `/etc/os-release`.
## Trust model
`image pull` delegates trust to the OCI registry the user selected.
`go-containerregistry` verifies layer digests against the manifest
during download, so a tampered mirror can't ship modified layers
without breaking the sha256 chain. Beyond that, banger does not
verify OCI image signatures (cosign/sigstore) — users who care should
verify their references out-of-band.
`image pull` (OCI path) delegates trust to the registry the user
selected. `go-containerregistry` verifies layer digests against the
manifest during download, so a tampered mirror can't ship modified
layers without breaking the sha256 chain. Banger does not verify OCI
image signatures (cosign/sigstore) — users who care should verify
references out-of-band.