diff --git a/AGENTS.md b/AGENTS.md index 25294ef..331062f 100644 --- a/AGENTS.md +++ b/AGENTS.md @@ -7,7 +7,9 @@ Always run `make build` before commit. - `cmd/banger` and `cmd/bangerd` are the main user entrypoints. - `internal/` contains the daemon, CLI, RPC, storage, Firecracker integration, guest helpers, and the experimental web UI. - `internal/daemon/` is the composition root; pure helpers live in its subpackages (`opstate`, `dmsnap`, `fcproc`, `imagemgr`, `session`, `workspace`). See `internal/daemon/ARCHITECTURE.md`. -- `scripts/` contains explicit manual helper workflows for rootfs and kernel preparation. +- `internal/imagecat/` and `internal/kernelcat/` embed the image + kernel catalogs. +- `images/golden/` is the Dockerfile for the `debian-bookworm` catalog entry. +- `scripts/` contains manual helper workflows for rootfs, kernel, and bundle preparation. - `build/bin/` is the canonical source-checkout build output. - `build/manual/` is the canonical source-checkout location for manual rootfs/kernel artifacts. @@ -17,19 +19,20 @@ Always run `make build` before commit. - `make test` runs `go test ./...`. - `make lint` runs `gofmt -l`, `go vet ./...`, and `shellcheck --severity=error` on `scripts/*.sh`. Run before commits. - `./build/bin/banger doctor` checks host readiness. -- `./build/bin/banger image build --from-image ` builds a managed image from an existing registered image. +- `./build/bin/banger vm run` is the primary user-facing entry point — auto-pulls the default image + kernel from the catalogs if missing. +- `./build/bin/banger image pull ` uses the bundle catalog (fast) when `` is a catalog entry, or falls through to the OCI path for arbitrary registry refs. See `docs/image-catalog.md` and `docs/oci-import.md`. - `./build/bin/banger image register ...` registers an unmanaged host-side image stack. +- `./build/bin/banger image build --from-image ` builds a managed image from an existing one. - `./build/bin/banger image promote ` copies an unmanaged image into daemon-owned managed artifacts. -- `scripts/make-generic-kernel.sh` builds a Firecracker-optimized vmlinux from upstream sources (no initrd, all drivers built-in). This is the recommended kernel for OCI-pulled images. -- `make void-kernel`, `make rootfs-void`, and `make void-register` drive the experimental Void flow under `./build/manual`. -- `scripts/publish-kernel.sh ` packages a locally-imported kernel and uploads it to the catalog; see `docs/kernel-catalog.md`. -- `banger image pull --kernel-ref ` pulls a rootfs from any OCI registry; see `docs/oci-import.md` (experimental — file-ownership caveat). +- `scripts/make-generic-kernel.sh` builds a Firecracker-optimized vmlinux from upstream sources. `scripts/publish-kernel.sh ` publishes it to the kernel catalog. +- `scripts/publish-golden-image.sh` rebuilds + publishes the golden image bundle and patches the image catalog. ## Image Model - Managed images own the full boot set: rootfs, optional work-seed, kernel, optional initrd, and optional modules. -- There is no runtime bundle and no auto-registered default image from disk paths. -- `default_image_name` selects a registered image only. +- The image catalog ships pre-built bundles. `vm run` auto-pulls the default catalog entry; `image pull ` can be invoked explicitly. +- `default_image_name` defaults to `debian-bookworm`. On miss, the daemon auto-pulls from `imagecat` before surfacing "not found". +- Kernel references follow the same auto-pull pattern against `kernelcat`. ## Config @@ -50,7 +53,7 @@ Always run `make build` before commit. ## Testing Guidance - Primary automated coverage is `go test ./...`. -- For lifecycle changes, smoke-test with `vm create`, `vm ssh`, `vm stop`, and `vm delete`. +- For lifecycle changes, smoke-test with `vm run` end-to-end (covers create + start + boot + ssh). - If guest provisioning changes, document whether existing images must be rebuilt or recreated. ## Security diff --git a/README.md b/README.md index 7a72c06..9a74565 100644 --- a/README.md +++ b/README.md @@ -1,330 +1,198 @@ # banger -`banger` manages Firecracker development VMs with a local daemon, managed image artifacts, and an experimental localhost web UI. +One-command development sandboxes on Firecracker microVMs. + +## Quick start + +```bash +make install +banger vm run --name sandbox +``` + +`banger vm run` auto-pulls the default golden image (Debian bookworm +with systemd, sshd, Docker CE, git, jq, mise, and the usual dev tools) +and kernel from the embedded catalog if they aren't already local, +creates a VM, starts it, and drops you into an interactive ssh +session. First run takes a couple minutes (bundle download); +subsequent `vm run`s are seconds. ## Requirements - Linux with `/dev/kvm` - `sudo` -- Firecracker installed on `PATH`, or `firecracker_bin` set in config -- The usual host tools checked by `./build/bin/banger doctor` +- Firecracker on `PATH`, or `firecracker_bin` set in config +- host tools checked by `banger doctor` -`banger` now owns complete managed image sets. A managed image includes: - -- `rootfs` -- optional `work-seed` -- `kernel` -- optional `initrd` -- optional `modules` - -There is no runtime bundle anymore. - -## Build - -```bash -make build -``` - -This writes: - -- `./build/bin/banger` -- `./build/bin/bangerd` -- `./build/bin/banger-vsock-agent` - -## Install +## Build + install ```bash make install ``` -That installs: +Installs: -- `banger` -- `bangerd` -- the `banger-vsock-agent` companion helper under `../lib/banger/` +- `banger` (CLI) +- `bangerd` (daemon, auto-starts on first CLI call) +- `banger-vsock-agent` (companion, under `$PREFIX/lib/banger/`) + +## `vm run` + +One command, three modes: + +```bash +banger vm run # bare sandbox — drops into ssh +banger vm run ./repo # workspace at /root/repo — drops into ssh +banger vm run ./repo -- make test # workspace + run command, exit with its status +``` + +- Bare mode gives you a clean shell. +- Workspace mode (with a path) copies the repo's tracked + untracked + non-ignored files into `/root/repo` and kicks off a best-effort + mise tooling bootstrap from the repo's `.mise.toml` / + `.tool-versions`. Log: `/root/.cache/banger/vm-run-tooling-.log`. +- Command mode (`-- `) runs the command in the guest; exit code + propagates through `banger`. + +Disconnecting from an interactive session leaves the VM running. Use +`vm stop` / `vm delete` to clean up. + +`--branch` and `--from` apply only to workspace mode. + +## Image catalog + +`banger image pull ` resolves `` in the embedded catalog +and fetches a pre-built bundle (rootfs.ext4 + manifest, tar+zstd). The +kernel referenced by the manifest auto-pulls too. `vm run` calls this +for you on demand. + +Today's catalog: + +| Name | Distro | Kernel | +|------|--------|--------| +| `debian-bookworm` | Debian 12 slim + sshd + docker + dev tools | `generic-6.12` | + +The catalog ships embedded in the banger binary. See +[`docs/image-catalog.md`](docs/image-catalog.md) for maintenance. + +## Power-user flows + +Skip this section if `vm run` is enough. + +### `vm create` — low-level primitive + +For scripting or `--no-start` provisioning: + +```bash +banger vm create --image debian-bookworm --name testbox --no-start +banger vm start testbox +banger vm ssh testbox +banger vm stop testbox +``` + +### `image pull ` — arbitrary container images + +For images outside the catalog, pull from any OCI registry: + +```bash +banger image pull docker.io/library/alpine:3.20 --kernel-ref generic-6.12 +``` + +Layers are flattened, ownership is fixed, banger's guest agents are +injected, and a first-boot service installs `openssh-server` via the +guest's package manager. See [`docs/oci-import.md`](docs/oci-import.md) +for supported distros and caveats. + +### `image register` — existing host-side stack + +```bash +banger image register --name base \ + --rootfs /abs/path/rootfs.ext4 \ + --kernel-ref generic-6.12 +``` + +### `image build --from-image` — derived images + +```bash +banger image build --name devbox --from-image debian-bookworm --docker +``` + +Spins up a transient VM from a base image, applies opinionated +customisation (mise, claude, pi, tmux plugins), saves a new managed +image. + +### Workspace + session primitives + +Long-lived guest commands managed by the daemon, attachable over a +local Unix socket bridge: + +```bash +banger vm workspace prepare ./other-repo --guest-path /root/repo +banger vm session start --name planner --cwd /root/repo --stdin-mode pipe -- pi --mode rpc +banger vm session attach planner +banger vm session logs planner --stream stderr +banger vm session stop planner +``` + +For ACP-aware host tooling: `banger vm acp ` bridges stdio to +guest `opencode acp` over SSH. ## Config -Config lives at `~/.config/banger/config.toml`. +Config lives at `~/.config/banger/config.toml`. All keys optional. -Supported keys: +Commonly set: -- `log_level` -- `web_listen_addr` -- `firecracker_bin` -- `ssh_key_path` -- `default_image_name` -- `auto_stop_stale_after` -- `stats_poll_interval` -- `metrics_poll_interval` -- `bridge_name` -- `bridge_ip` -- `cidr` -- `tap_pool_size` -- `default_dns` +- `default_image_name` — image to use when `--image` is omitted + (defaults to `debian-bookworm`, auto-pulled from the catalog if not + local). +- `ssh_key_path` — host SSH key; if unset banger creates + `~/.config/banger/ssh/id_ed25519`. +- `firecracker_bin` — override the auto-resolved `PATH` lookup. +- `web_listen_addr` — experimental web UI (default + `127.0.0.1:7777`; set to `""` to disable). +- Network: `bridge_name`, `bridge_ip`, `cidr`, `tap_pool_size`, + `default_dns`. -If `ssh_key_path` is unset, banger creates and uses: +Full key list in `internal/config/config.go`. -- `~/.config/banger/ssh/id_ed25519` +## Credential sync -`default_image_name` now only means “use this registered image when `vm create` omits `--image`”. The daemon does not auto-register images from host paths. +If these host auth files exist, banger syncs them into the guest at +VM start: -## Core Workflow +| Host | Guest | +|------|-------| +| `~/.local/share/opencode/auth.json` | `/root/.local/share/opencode/auth.json` | +| `~/.claude/.credentials.json` | `/root/.claude/.credentials.json` | +| `~/.pi/agent/auth.json` | `/root/.pi/agent/auth.json` | -Check the host: - -```bash -./build/bin/banger doctor -``` - -Register an existing host-side image stack: - -```bash -./build/bin/banger image register \ - --name base \ - --rootfs /abs/path/rootfs.ext4 \ - --kernel /abs/path/vmlinux \ - --initrd /abs/path/initrd.img \ - --modules /abs/path/modules -``` - -Or pull a pre-built kernel from the catalog and reference it by name: - -```bash -./build/bin/banger kernel list --available -./build/bin/banger kernel pull generic-6.12 -./build/bin/banger image register \ - --name base \ - --rootfs /abs/path/rootfs.ext4 \ - --kernel-ref generic-6.12 -``` - -See [`docs/kernel-catalog.md`](docs/kernel-catalog.md) for catalog -maintenance. - -Or pull a rootfs directly from any OCI registry (Docker Hub, GHCR, …): - -```bash -./build/bin/banger image pull docker.io/library/debian:bookworm \ - --kernel-ref generic-6.12 -``` - -`image pull` downloads the image, flattens its layers into an ext4 -rootfs, applies tar-header ownership via debugfs, and pre-injects -banger's guest agents (vsock agent + network bootstrap + a first-boot -unit that installs `openssh-server` via the guest's native package -manager). Boots as a banger VM directly, no `image build` step -required. See [`docs/oci-import.md`](docs/oci-import.md) for -supported distros and current limitations. - -Build a managed image from an existing registered image: - -```bash -./build/bin/banger image build \ - --name devbox \ - --from-image base \ - --docker -``` - -Promote an unmanaged image into daemon-owned managed artifacts: - -```bash -./build/bin/banger image promote base -``` - -Spin up a sandbox VM and drop straight into it: - -```bash -./build/bin/banger vm run # bare sandbox, interactive ssh -./build/bin/banger vm run ../some-repo # workspace at /root/repo, interactive ssh -./build/bin/banger vm run ../some-repo -- make test # workspace, run command, exit with its status -``` - -`vm run` creates a VM, prepares a workspace if you pass a path, and then either drops you into an interactive ssh session or runs the `--`-delimited command to completion. The command's exit code propagates through `banger`. Disconnecting from the interactive session leaves the VM running; use `vm stop` / `vm delete` to clean up. - -When you pass a path, `vm run` copies a git checkout plus tracked and untracked non-ignored files into `/root/repo`, then kicks off a best-effort `mise` tooling bootstrap that runs asynchronously inside the guest (log at `/root/.cache/banger/vm-run-tooling-.log`). The bootstrap is skipped in bare and command modes. Flags like `--branch` and `--from` require a path. - -For scripting or lower-level control, `vm create` remains available as a primitive (use `--no-start` when you just want to provision): - -```bash -./build/bin/banger vm create --image devbox --name testbox --no-start -./build/bin/banger vm start testbox -./build/bin/banger vm ssh testbox -./build/bin/banger vm stop testbox -``` - -`vm create` stays synchronous by default, but on a TTY it now shows live progress until the VM is fully ready. - -For ACP-aware host tools, `./build/bin/banger vm acp ` bridges stdio to guest `opencode acp` over SSH. It uses `/root/repo` when that checkout exists, otherwise `/root`, and `--cwd` lets you override the guest working directory explicitly. - -If you want reusable orchestration primitives instead of the `vm run` convenience flow, use the daemon-backed workspace and session commands directly: - -```bash -./build/bin/banger vm workspace prepare -./build/bin/banger vm workspace prepare ../other-repo --guest-path /root/repo --readonly -./build/bin/banger vm session start --name planner --cwd /root/repo --stdin-mode pipe -- pi --mode rpc --no-session -./build/bin/banger vm session list -./build/bin/banger vm session attach planner -./build/bin/banger vm session logs planner --stream stderr -./build/bin/banger vm session stop planner -``` - -`vm workspace prepare` materializes a local git checkout into a running VM. The default guest path is `/root/repo` and the default mode is a shallow metadata copy plus tracked and untracked non-ignored overlay. Repositories with git submodules must use `--mode full_copy`; the metadata-based modes still reject them. - -`vm session start` creates a daemon-managed long-lived guest command. The daemon preflights that the requested guest `cwd` exists and that the main command, plus any repeated `--require-command` entries, exist in guest `PATH` before launch. Use `--stdin-mode pipe` when you need live `attach`; otherwise use the default detached mode and inspect sessions with `list`, `show`, `logs`, `stop`, and `kill`. - -`vm session attach` is currently exclusive and same-host only. The daemon exposes a local Unix socket bridge using `stdio_mux_v1`, so only one active attach is allowed at a time. Pipe-mode sessions keep enough guest-side state for the daemon to rebuild that bridge after a daemon restart. +Host-side changes take effect after the VM restarts. Session/history +directories are not copied. ## Web UI (experimental) -`bangerd` serves an experimental local web UI by default at: - -- `http://127.0.0.1:7777` - -The UI is convenient for local observability but is **not a stable or -supported interface**. Its endpoints, layout, and behaviour may change -without notice, and it has not been hardened for anything beyond single-user -localhost use. Do not expose the listen address to a shared network. - -See the effective URL with: - -```bash -./build/bin/banger daemon status -``` - -Disable it with: - -```toml -web_listen_addr = "" -``` - -## Guest Services - -Provisioned glibc-backed images include: - -- `banger-vsock-agent` -- guest networking bootstrap -- `mise` -- `opencode` -- `claude` -- `pi` -- a default guest `opencode` service on `0.0.0.0:4096` - -Alpine currently remains `opencode`-only. - -If these host auth files exist, `banger` syncs them into the guest on VM start: - -- `~/.local/share/opencode/auth.json` -> `/root/.local/share/opencode/auth.json` -- `~/.claude/.credentials.json` -> `/root/.claude/.credentials.json` -- `~/.pi/agent/auth.json` -> `/root/.pi/agent/auth.json` - -Changes on the host take effect after the VM is restarted. Session/history directories are not copied. - -From the host: - -```bash -./build/bin/banger vm ports testbox -opencode attach http://:4096 -``` - -## Manual Helpers - -The shell helpers are now explicit manual workflows under `./build/manual`. - -Rebuild a Debian-style manual rootfs: - -```bash -make rootfs ARGS='--base-rootfs /abs/path/rootfs.ext4 --kernel /abs/path/vmlinux --initrd /abs/path/initrd.img --modules /abs/path/modules' -``` - -The output lands in: - -- `./build/manual/rootfs-docker.ext4` -- `./build/manual/rootfs-docker.work-seed.ext4` - -## Experimental Void Flow - -Stage a Void kernel: - -```bash -make void-kernel -``` - -Build the experimental Void rootfs: - -```bash -make rootfs-void -``` - -Register it: - -```bash -make void-register -``` - -That flow uses: - -- `./build/manual/void-kernel/` -- `./build/manual/rootfs-void.ext4` -- `./build/manual/rootfs-void.work-seed.ext4` - -## Experimental Alpine Flow - -Stage an Alpine virt kernel: - -```bash -make alpine-kernel -``` - -Build the experimental Alpine rootfs: - -```bash -make rootfs-alpine -``` - -Register it: - -```bash -make alpine-register -``` - -Create a VM from it: - -```bash -./build/bin/banger vm create --image alpine --name alpine-dev -``` - -That flow uses: - -- `./build/manual/alpine-kernel/` -- `./build/manual/rootfs-alpine.ext4` -- `./build/manual/rootfs-alpine.work-seed.ext4` - -The experimental Alpine flow stages a pinned Alpine release by default. Override -that pin with `ALPINE_RELEASE=...` when running the `make alpine-kernel` and -`make rootfs-alpine` helpers if you need a different patch release. - -Alpine support currently applies to the explicit register-and-run flow above. -The generic `banger image build --from-image ...` path remains Debian/systemd- -oriented and should not be treated as an Alpine image builder. +`bangerd` serves a local web UI at `http://127.0.0.1:7777` by default. +Convenient for local observability, **not a stable interface**. Do +not expose the listen address to a shared network. ## Security -Guest VMs are single-user development sandboxes, not multi-tenant servers. -Every provisioned image is configured with: +Guest VMs are single-user development sandboxes, not multi-tenant +servers. Every provisioned image is configured with: ``` PermitRootLogin yes StrictModes no ``` -This is intentional. The host SSH key is the only authentication mechanism, -no password auth is enabled, and VMs are reachable only through the host -bridge network (`172.16.0.0/24` by default). Do not expose the bridge -interface or the VM guest IPs to an untrusted network. +The host SSH key is the only authentication mechanism, no password +auth is enabled, and VMs are reachable only through the host bridge +network (`172.16.0.0/24` by default). Do not expose the bridge +interface or guest IPs to an untrusted network. ## Notes -- Firecracker is resolved from `PATH` by default. - Managed image delete removes the daemon-owned artifact dir. -- The companion vsock helper is internal to the install/build layout, not a user-configured runtime path. +- Layer blob cache for OCI pulls lives under `~/.cache/banger/oci/`. +- Image bundle cache doesn't exist — bundles are extracted directly + into the image store; re-pulls download fresh. diff --git a/docs/image-catalog.md b/docs/image-catalog.md new file mode 100644 index 0000000..a0d81ac --- /dev/null +++ b/docs/image-catalog.md @@ -0,0 +1,123 @@ +# Image catalog + +The image catalog ships pre-built banger rootfs bundles so users don't +have to register or build anything. It's the fast path behind +`banger vm run` (auto-pull) and `banger image pull `. The +catalog is embedded into the banger binary and updated each release. + +End-user flow: + +```bash +banger image pull debian-bookworm # explicit +banger vm run --name sandbox # implicit (auto-pulls) +``` + +## Architecture + +Two parts — the same shape as the kernel catalog: + +1. **`internal/imagecat/catalog.json`** — JSON manifest embedded into + the banger binary via `go:embed`. Each entry: name, distro, arch, + kernel_ref (a `kernelcat` entry name), tarball URL, tarball + sha256, size. + +2. **Tarballs at `https://images.thaloco.com/`** — Cloudflare R2 + bucket `banger-images`, fronted by a public custom domain. Each + tarball is `--.tar.zst` (content- + addressed filename so CDN edge cache can never serve stale bytes + for the URL the catalog points at). Contents at the archive root: + `rootfs.ext4` (finalized: flattened + ownership-fixed + agent- + injected at build time) and `manifest.json`. + +The `banger image pull` bundle path streams the tarball, verifies +sha256 against the catalog entry, extracts both files into a staging +dir, resolves the kernel via `kernel_ref` (auto-pulling from +`kernelcat` if the user hasn't pulled it yet), stages boot artifacts +alongside, and registers the result as a managed image. + +The same `image pull` command transparently falls through to the +existing OCI-pull path when `` doesn't match a catalog entry — +see [`docs/oci-import.md`](oci-import.md). + +## Adding or updating an entry + +The repo has no CI for bundle publishing yet. Catalog updates are +manual. + +```bash +# 1. Build the bundle + upload + patch catalog.json in one shot. +scripts/publish-golden-image.sh + +# 2. Review and commit the catalog change. +git diff -- internal/imagecat/catalog.json +git add internal/imagecat/catalog.json +git commit -m 'imagecat: publish debian-bookworm' + +# 3. Rebuild so the new catalog is embedded. +make build +``` + +`scripts/publish-golden-image.sh` wraps `scripts/make-golden-bundle.sh` +(which runs `docker build` on `images/golden/Dockerfile` then pipes +`docker export` into `banger internal make-bundle`), computes the +bundle's sha256, uses the first 12 hex chars as a cache-busting +filename suffix, uploads via `rclone` to R2, HEAD-checks the public +URL, and patches `internal/imagecat/catalog.json`. + +Environment overrides if the defaults need to change: +`RCLONE_REMOTE`, `RCLONE_BUCKET`, `BASE_URL`. + +`--skip-upload` builds the bundle into `dist/` and stops — useful for +local testing without touching R2 or the catalog. + +## Bundle format + +A bundle is a tar+zstd archive with exactly two entries at the root: + +``` +rootfs.ext4 # finalized banger rootfs +manifest.json # {name, distro, arch, kernel_ref, description} +``` + +`rootfs.ext4` is fully prepared at build time: ownership fixed via +`debugfs sif`, banger guest agents (vsock agent, network bootstrap, +first-boot unit) already injected and enabled in +`multi-user.target.wants`. The pull path only has to place the file +and register the image — no mkfs, no ownership pass, no injection on +the daemon host. + +## Removing an entry + +1. Remove the entry from `internal/imagecat/catalog.json` and commit. +2. Delete the tarball from R2: + `rclone delete banger-images:banger-images/--.tar.zst`. +3. Rebuild banger. + +Already-pulled local images are not invalidated — users keep using +them until they run `banger image delete `. + +## Versioning conventions + +- **Entry names**: `-` (e.g. `debian-bookworm`). + Per-release names make it trivial to publish `debian-trixie` + alongside without collisions. +- **Content-addressed filenames**: the `-` suffix is + mandatory (set by `publish-golden-image.sh`). Never reuse a URL for + different bytes. +- **Architecture**: `x86_64` only today. The `arch` field is additive + — adding `arm64` is a config change, not a schema change. + +## Trust model + +Same as the kernel catalog: the embedded `catalog.json` carries each +bundle's sha256, and `imagecat.Fetch` rejects any download whose hash +doesn't match. This protects against transport corruption and against +an attacker swapping an R2 object without landing a commit in the +banger repo. GPG/sigstore signing is deferred until banger is public +and the threat model justifies the operational overhead. + +## Hosting + +Tarballs live in Cloudflare R2 (bucket `banger-images`), served at +`images.thaloco.com`. The bucket is publicly readable; writes require +the R2 API token configured on the `banger-images` rclone remote. diff --git a/docs/oci-import.md b/docs/oci-import.md index 43aeb7d..829fc84 100644 --- a/docs/oci-import.md +++ b/docs/oci-import.md @@ -1,193 +1,153 @@ # OCI import (`banger image pull`) -`banger image pull ` downloads a container image from any -OCI-compatible registry (Docker Hub, GHCR, quay.io, self-hosted, …), -flattens its layers into an ext4 rootfs, and registers the result as -a managed banger image. +`banger image pull` has two paths. The primary one — catalog bundle — +is documented in [`docs/image-catalog.md`](image-catalog.md). This +doc covers the fallthrough: OCI-registry pull for arbitrary container +images. -Paired with the kernel catalog, this dissolves the "where do I get a -rootfs" bottleneck for most users — any distro that ships an official -container image can now boot (eventually) as a banger VM. +## When to use it + +Use the OCI path when you need a distro or image that isn't in the +catalog. The catalog covers the common happy path +(`debian-bookworm`); anything else (`alpine`, `fedora`, `ubuntu`, +custom corporate images) goes through OCI pull. ```bash -banger kernel pull void-6.12 -banger image pull docker.io/library/debian:bookworm --kernel-ref void-6.12 -banger image list # debian-bookworm appears, Managed=true +banger image pull docker.io/library/alpine:3.20 --kernel-ref generic-6.12 +banger image pull ghcr.io/myorg/devimg:v2 --kernel-ref generic-6.12 ``` +`banger image pull` dispatches based on the reference: + +- `banger image pull debian-bookworm` → catalog (fast path). +- `banger image pull docker.io/library/foo:bar` → OCI (anything not + in the catalog). + ## What works -- Pulling any public OCI image that exposes a `linux/amd64` manifest. +- Any public OCI image that exposes a `linux/amd64` manifest. - Correct layer replay with whiteout semantics (`.wh.*` deletes, `.wh..wh..opq` opaque-dir markers). - Path-traversal and relative-symlink-escape protection. -- Content-aware default sizing (`content × 1.25`, floor 1 GiB). -- Layer caching on disk, keyed by blob SHA256. -- **File ownership preservation.** Tar-header uid/gid/mode is captured - during flatten and applied to the resulting ext4 via a `debugfs` - pass, so setuid binaries (`sudo`, `passwd`) and root-owned config - files (`/etc/shadow`, `/etc/sudoers`) end up correctly owned. -- **Banger guest agents pre-injected.** The pulled ext4 ships with - `/usr/local/bin/banger-vsock-agent`, `banger-network.service`, and - `banger-vsock-agent.service` already in place and enabled. -- **First-boot sshd install.** A one-shot systemd service installs - `openssh-server` via the guest's package manager on first boot — - apt-get / apk / dnf / pacman / zypper dispatch based on - `/etc/os-release`. Subsequent boots skip the install. -- Piping pulled images into the existing `banger image build - --from-image` flow. +- Content-aware default sizing (`content × 1.5`, floor 1 GiB). +- Layer caching on disk, keyed by blob sha256. +- **Ownership preservation** — tar-header uid/gid/mode captured + during flatten, applied to the ext4 via a `debugfs` pass, so + setuid binaries (`sudo`, `passwd`) and root-owned config + (`/etc/shadow`, `/etc/sudoers`) end up correctly owned. +- **Pre-injected banger agents** — the pulled ext4 ships with + `banger-vsock-agent`, `banger-network.service`, and the + `banger-first-boot` unit already enabled. +- **First-boot sshd install** — a one-shot systemd service installs + `openssh-server` via the guest's package manager on first boot. + Dispatches on `/etc/os-release` → `apt-get` / `apk` / `dnf` / + `pacman` / `zypper`. Subsequent boots skip the install. +- Composition with `image build --from-image`. ## What doesn't yet work -- **Private registries**. Auth is not implemented; anonymous pulls - only. Docker Hub, GHCR (public), quay.io (public), etc. all work. -- **Non-`linux/amd64` platforms**. The kernel catalog is x86_64-only, - so pulled rootfses match. `arm64` is additive in the schema; wire- - up lands when a user needs it. -- **Non-systemd distros.** The injected units assume systemd as PID 1. - Alpine ≥3.20 ships systemd; older alpine + void + busybox-init - images won't honour the banger-network / banger-first-boot units. -- **First boot needs network access.** The provisioning step reaches - out to the distro's package repo to install openssh-server. VMs - without NAT or without the bridge reaching the internet will time - out on first boot. The marker file stays in place so a later boot - retries. +- **Private registries**. Anonymous pulls only. Docker Hub, GHCR + (public), quay.io (public) all work. Adding auth via + `authn.DefaultKeychain` (from `go-containerregistry`) is a cheap + follow-up when someone needs it. +- **Non-`linux/amd64`**. The kernel catalog is x86_64-only, so pulled + rootfses match. `arm64` is additive in the schema. +- **Non-systemd rootfses**. The injected units assume systemd as + PID 1. Alpine ≥3.20 ships systemd; older alpine + void + busybox- + init images won't honour the banger-* units. +- **First boot needs network access**. The first-boot sshd install + reaches out to the distro's package repo. VMs without NAT or + without the bridge reaching the internet time out. The marker file + stays in place so a later restart retries. ## Architecture -`internal/imagepull/` owns the pure mechanics: +`internal/imagepull/` owns the mechanics: -- **`Pull`** (`imagepull.go`) wraps `go-containerregistry`'s - `remote.Image` with the `linux/amd64` platform pinned. Layer - blobs are cached on disk via `cache.NewFilesystemCache` under - `/blobs/` — Pull itself does not drain the layer - streams; that happens lazily during `Flatten`, and the cache - populates on read. -- **`Flatten`** (`flatten.go`) replays layers oldest-first into a - staging directory, applying whiteouts and rejecting unsafe paths. - Returns a `Metadata` map capturing per-file uid/gid/mode from - each tar header. -- **`BuildExt4`** (`ext4.go`) runs `mkfs.ext4 -F -d - -E root_owner=0:0` to populate the image file at create time — - no mount, no sudo, no loopback. Requires `e2fsprogs ≥ 1.43` - (`mkfs.ext4 -d` is the populate-at-create flag; nearly all - modern distros ship it). -- **`ApplyOwnership`** (`ownership.go`) streams a batched - `set_inode_field` script to `debugfs -w -f -` to rewrite per-file - uid/gid/mode to the captured tar-header values. Without this pass - the ext4 would carry the runner's on-disk uids. -- **`InjectGuestAgents`** (`inject.go`) uses the same `debugfs` - scripting to drop banger's guest-side assets into the pulled ext4 - with root ownership: - - `/usr/local/bin/banger-vsock-agent` - - `/usr/local/libexec/banger-network-bootstrap` - - `/usr/local/libexec/banger-first-boot` - - `/etc/systemd/system/banger-{network,vsock-agent,first-boot}.service` - - enable-at-boot symlinks under `multi-user.target.wants/` - - `/etc/modules-load.d/banger-vsock.conf` - - `/var/lib/banger/first-boot-pending` (marker file) +- **`Pull`** wraps `go-containerregistry`'s `remote.Image` with the + `linux/amd64` platform pinned. Layer blobs cache under + `~/.cache/banger/oci/blobs/` and populate lazily during flatten. +- **`Flatten`** replays layers oldest-first into a staging directory, + applies whiteouts, rejects unsafe paths. Returns a `Metadata` map + of per-file uid/gid/mode from tar headers. +- **`BuildExt4`** runs `mkfs.ext4 -F -d -E root_owner=0:0` + at the size of the pre-truncated file — no mount, no sudo, no + loopback. Requires `e2fsprogs ≥ 1.43`. +- **`ApplyOwnership`** streams a batched `set_inode_field` script to + `debugfs -w` to rewrite per-file uid/gid/mode to the captured tar- + header values. +- **`InjectGuestAgents`** uses the same `debugfs` scripting to drop + banger's guest assets into the ext4 with root ownership: + vsock agent binary, network bootstrap + unit, first-boot script + + unit, `multi-user.target.wants` symlinks, vsock modules-load + config, `/var/lib/banger/first-boot-pending` marker. -`internal/daemon/images_pull.go` orchestrates: +`internal/daemon/images_pull.go` orchestrates `pullFromOCI`: -1. Parse + validate the OCI ref. -2. Derive a friendly default name (`debian-bookworm` for - `docker.io/library/debian:bookworm`) when `--name` is omitted. -3. Resolve kernel info via the shared `resolveKernelInputs` helper - (the same code path as `image register --kernel-ref`). -4. Stage at `/.staging`; extract layers to a temp - tree under `os.TempDir` (bulk transient data stays off the - persistent state filesystem). -5. `imagepull.BuildExt4` produces `/rootfs.ext4`. -6. `ApplyOwnership` + `InjectGuestAgents` run in one finalize step. -7. `imagemgr.StageBootArtifacts` stages the kernel triple alongside. -8. Atomic `os.Rename(, )` publishes the artifact dir. -9. Persist a `model.Image{Managed: true, …}` record. - -Any failure removes the staging dir. Post-rename failures remove the -final dir and roll back the store write. +1. Parse + validate the OCI ref, derive a default name when `--name` + is omitted (`debian-bookworm` from + `docker.io/library/debian:bookworm`). +2. Resolve kernel info via `resolveKernelInputs` (auto-pulls from + `kernelcat` if `--kernel-ref` names a catalog entry that isn't + yet local). +3. Stage at `/.staging`; extract layers to a temp + tree under `$TMPDIR`. +4. `BuildExt4` → `ApplyOwnership` → `InjectGuestAgents`. +5. `imagemgr.StageBootArtifacts` stages the kernel triple alongside. +6. Atomic `os.Rename` publishes the artifact dir. +7. Persist a `model.Image{Managed: true, …}` record. ## Guest-side boot sequence -On the first boot of a pulled image, systemd starts three banger -units in order: +On first boot of a pulled image: -1. **`banger-network.service`** — runs the bootstrap script that - parses `/etc/banger-network.conf` (written by banger's VM-create - lifecycle) and brings the guest interface up with the assigned IP. -2. **`banger-first-boot.service`** (only on first boot; removes its - own trigger file on success) — reads `/etc/os-release`, dispatches - to the native package manager, installs `openssh-server`, enables - `ssh.service` / `sshd.service`. -3. **`banger-vsock-agent.service`** — runs the health-check daemon - banger uses to confirm the VM is alive. +1. **`banger-network.service`** — brings the guest interface up with + the IP assigned by banger's VM-create lifecycle. +2. **`banger-first-boot.service`** (first boot only) — reads + `/etc/os-release`, dispatches to the native package manager, + installs `openssh-server`, enables `ssh.service`. +3. **`banger-vsock-agent.service`** — the health-check daemon banger + uses to confirm the VM is alive. -After first boot completes, subsequent boots skip the install step -entirely. Banger's host-side SSH polling (`guest.WaitForSSH`) -naturally retries until sshd is listening. +Subsequent boots skip step 2. -## Adding distro support +## Adding distro support to first-boot `internal/imagepull/assets/first-boot.sh` is the POSIX-sh dispatch. -Add a new `ID=` branch and its install command to the `case` block, -then rebuild banger — the asset is `go:embed`-ed into the binary. +Add a new `ID=` branch and its install command, then rebuild banger +(the asset is `go:embed`-ed). + Supported `ID` values today: `debian`, `ubuntu`, `kali`, `raspbian`, `linuxmint`, `pop`, `alpine`, `fedora`, `rhel`, `centos`, `rocky`, `almalinux`, `arch`, `archlinux`, `manjaro`, `opensuse*`, `suse`. -Unknown distros fall back to `ID_LIKE`, then error clearly with a -pointer to edit the script. +Unknown distros fall back to `ID_LIKE`, then error cleanly. ## Paths -| What | Where | Purpose | -|------|-------|---------| -| Layer blob cache | `~/.cache/banger/oci/blobs/sha256/` | Re-pulls of the same image digest are local-only | -| Staging dir | `~/.local/state/banger/images/.staging/` | Short-lived; atomic-renamed to `/` on success | -| Staging rootfs tree | `$TMPDIR/banger-pull-/` | Extraction scratch space; removed after ext4 build | -| Published image | `~/.local/state/banger/images//rootfs.ext4` | Managed artifact stored alongside the kernel triple | - -## Composition with `image build` - -A pulled image boots as-is — ownership is correct, sshd installs on -first boot, banger's agents are in place. That means the existing -`image build --from-image` pipeline composes on top: - -```bash -banger image build --from-image debian-bookworm --name debian-dev --docker -``` - -`image build` spins up a transient VM using the base image, runs -`scripts/customize.sh` over it, and saves the result as a new managed -image with the opinionated tooling (mise, opencode, claude, pi, tmux -plugins, optionally docker) layered on top. +| What | Where | +|------|-------| +| Layer blob cache | `~/.cache/banger/oci/blobs/sha256/` | +| Staging dir | `~/.local/state/banger/images/.staging/` | +| Extraction scratch | `$TMPDIR/banger-pull-/` | +| Published image | `~/.local/state/banger/images//rootfs.ext4` | ## Tech debt -- **Auth**. When we add private-registry support, the natural path is - `authn.DefaultKeychain` from `go-containerregistry`, which already - honours `~/.docker/config.json` and the standard credential - helpers. No banger-specific config needed. - -- **Cache eviction**. Layer blobs under `OCICacheDir` accumulate - forever. A `banger image cache prune` command is a cheap follow-up - when disk usage becomes a complaint. - -- **First-boot timeout UX**. If you run `banger vm ssh` immediately - after `banger vm create`, the package install for `openssh-server` - may still be running and SSH will fail. Current mitigation: retry. - Better: a per-image `FirstBootPending` flag that tells the daemon - to extend its SSH wait timeout for the first boot, cleared on - success. Tracked but not implemented. - -- **Non-systemd distros**. The guest agents assume systemd. Adding +- **Auth**. When we add private-registry support, the natural path + is `authn.DefaultKeychain`, which honours `~/.docker/config.json` + and the standard credential helpers. +- **Cache eviction**. OCI layer blobs accumulate forever. A `banger + image cache prune` command is a cheap follow-up when disk usage + becomes a complaint. +- **Non-systemd rootfses**. The guest agents assume systemd. Adding openrc / s6 / busybox-init variants means keeping parallel unit - trees in `inject.go` keyed on `/etc/os-release`. Only pick up - when a user actually wants it. + trees keyed on `/etc/os-release`. ## Trust model -`image pull` delegates trust to the OCI registry the user selected. -`go-containerregistry` verifies layer digests against the manifest -during download, so a tampered mirror can't ship modified layers -without breaking the sha256 chain. Beyond that, banger does not -verify OCI image signatures (cosign/sigstore) — users who care should -verify their references out-of-band. +`image pull` (OCI path) delegates trust to the registry the user +selected. `go-containerregistry` verifies layer digests against the +manifest during download, so a tampered mirror can't ship modified +layers without breaking the sha256 chain. Banger does not verify OCI +image signatures (cosign/sigstore) — users who care should verify +references out-of-band.