thaloco/banger
1
Fork 0
Code Pull requests Releases Packages Activity Actions
banger/docs/oci-import.md
Thales Maciel 4d8dca6b72
image: add banger image cache prune for OCI cache cleanup 
OCI layer blobs accumulate forever — every pull writes layers to
~/.cache/banger/oci/blobs/sha256/<hex> via go-containerregistry's
filesystem cache, and nothing ever evicts them. The cache is purely
a re-pull-avoidance (every flattened image is independent of the
blobs that sourced it), so it's a perfect candidate for an opt-in
operator-driven prune.

New surface:
  * api: ImageCachePruneParams{DryRun}, ImageCachePruneResult
    {BytesFreed, BlobsFreed, DryRun, CacheDir}.
  * daemon: ImageService.PruneOCICache walks layout.OCICacheDir for
    a (bytes, blobs) tally, then — outside dry-run — atomically
    renames the cache aside, recreates it empty, and rm -rf's the
    aside dir. The rename-then-rm avoids leaving the cache in a
    half-removed state if a pull starts mid-prune (the in-flight
    pull's open files survive the rename via standard Linux
    semantics; it just sees a fresh empty cache afterwards). Missing
    cache dir is treated as zero — fresh installs that have never
    pulled an OCI image don't error.
  * dispatch: image.cache.prune RPC (paramHandler-wrapped, mirroring
    every other image RPC). Documented-methods test list updated.
  * cli: `banger image cache` group with a `prune` subcommand
    (--dry-run flag). Output is a single line: "freed 1.2 GiB
    across 47 blob(s) in /var/cache/banger/oci" or "would free …".
    formatBytes helper for the size pretty-print.

docs/oci-import.md: replaced the "Tech debt: cache eviction" bullet
with a "Cache lifecycle" section describing the new command and
the in-flight-pull caveat.

Tests: PruneOCICache covers the happy path (real prune empties the
cache, recreates an empty dir, doesn't leak the .pruning- aside),
the dry-run path (returns size, leaves blobs intact), and the
fresh-install path (cache dir absent → zero result, no error).
Smoke at JOBS=4 still green; live exercise against an empty cache
on a system install prints the expected zero summary.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
2026-04-28 16:32:57 -03:00

6.9 KiB
Raw Blame History

OCI import (banger image pull)

banger image pull has two paths. The primary one — catalog bundle — is documented in docs/image-catalog.md. This doc covers the fallthrough: OCI-registry pull for arbitrary container images.

When to use it

Use the OCI path when you need a distro or image that isn't in the catalog. The catalog covers the common happy path (debian-bookworm); anything else (alpine, fedora, ubuntu, custom corporate images) goes through OCI pull.

banger image pull docker.io/library/alpine:3.20 --kernel-ref generic-6.12
banger image pull ghcr.io/myorg/devimg:v2        --kernel-ref generic-6.12

banger image pull dispatches based on the reference:

  • banger image pull debian-bookworm → catalog (fast path).
  • banger image pull docker.io/library/foo:bar → OCI (anything not in the catalog).

What works

  • Any public OCI image that exposes a linux/amd64 manifest.
  • Correct layer replay with whiteout semantics (.wh.* deletes, .wh..wh..opq opaque-dir markers).
  • Path-traversal, debugfs-hostile filename, and relative-symlink-escape protection.
  • Content-aware default sizing (content × 1.5, floor 1 GiB).
  • Layer caching on disk, keyed by blob sha256.
  • Ownership preservation — tar-header uid/gid/mode captured during flatten, applied to the ext4 via a debugfs pass, so setuid binaries (sudo, passwd) and root-owned config (/etc/shadow, /etc/sudoers) end up correctly owned.
  • Pre-injected banger agents — the pulled ext4 ships with banger-vsock-agent, banger-network.service, and the banger-first-boot unit already enabled.
  • First-boot sshd install — a one-shot systemd service installs openssh-server via the guest's package manager on first boot. Dispatches on /etc/os-release → apt-get / apk / dnf / pacman / zypper. Subsequent boots skip the install.

What doesn't yet work

  • Private registries. Anonymous pulls only. Docker Hub, GHCR (public), quay.io (public) all work. Adding auth via authn.DefaultKeychain (from go-containerregistry) is a cheap follow-up when someone needs it.
  • Non-linux/amd64. The kernel catalog is x86_64-only, so pulled rootfses match. arm64 is additive in the schema.
  • Non-systemd rootfses. The injected units assume systemd as PID 1. Alpine ≥3.20 ships systemd; older alpine + void + busybox- init images won't honour the banger-* units.
  • First boot needs network access. The first-boot sshd install reaches out to the distro's package repo. VMs without NAT or without the bridge reaching the internet time out. The marker file stays in place so a later restart retries.

Architecture

internal/imagepull/ owns the mechanics:

  • Pull wraps go-containerregistry's remote.Image with the linux/amd64 platform pinned. Layer blobs cache under ~/.cache/banger/oci/blobs/ and populate lazily during flatten.
  • Flatten replays layers oldest-first into a staging directory, applies whiteouts, rejects unsafe paths plus filenames that banger's debugfs ownership fixup cannot encode safely. Returns a Metadata map of per-file uid/gid/mode from tar headers.
  • BuildExt4 runs mkfs.ext4 -F -d <staging> -E root_owner=0:0 at the size of the pre-truncated file — no mount, no sudo, no loopback. Requires e2fsprogs ≥ 1.43.
  • ApplyOwnership streams a batched set_inode_field script to debugfs -w to rewrite per-file uid/gid/mode to the captured tar- header values.
  • InjectGuestAgents uses the same debugfs scripting to drop banger's guest assets into the ext4 with root ownership: vsock agent binary, network bootstrap + unit, first-boot script + unit, multi-user.target.wants symlinks, vsock modules-load config, /var/lib/banger/first-boot-pending marker.

internal/daemon/images_pull.go orchestrates pullFromOCI:

  1. Parse + validate the OCI ref, derive a default name when --name is omitted (debian-bookworm from docker.io/library/debian:bookworm).
  2. Resolve kernel info via resolveKernelInputs (auto-pulls from kernelcat if --kernel-ref names a catalog entry that isn't yet local).
  3. Stage at <ImagesDir>/<id>.staging; extract layers to a temp tree under $TMPDIR.
  4. BuildExt4 → ApplyOwnership → InjectGuestAgents.
  5. imagemgr.StageBootArtifacts stages the kernel triple alongside.
  6. Atomic os.Rename publishes the artifact dir.
  7. Persist a model.Image{Managed: true, …} record.

Guest-side boot sequence

On first boot of a pulled image:

  1. banger-network.service — brings the guest interface up with the IP assigned by banger's VM-create lifecycle.
  2. banger-first-boot.service (first boot only) — reads /etc/os-release, dispatches to the native package manager, installs openssh-server, enables ssh.service.
  3. banger-vsock-agent.service — the health-check daemon banger uses to confirm the VM is alive.

Subsequent boots skip step 2.

Adding distro support to first-boot

internal/imagepull/assets/first-boot.sh is the POSIX-sh dispatch. Add a new ID= branch and its install command, then rebuild banger (the asset is go:embed-ed).

Supported ID values today: debian, ubuntu, kali, raspbian, linuxmint, pop, alpine, fedora, rhel, centos, rocky, almalinux, arch, archlinux, manjaro, opensuse*, suse. Unknown distros fall back to ID_LIKE, then error cleanly.

Paths

What Where
Layer blob cache ~/.cache/banger/oci/blobs/sha256/<hex>
Staging dir ~/.local/state/banger/images/<id>.staging/
Extraction scratch $TMPDIR/banger-pull-<rand>/
Published image ~/.local/state/banger/images/<id>/rootfs.ext4

Cache lifecycle

OCI layer blobs accumulate as you pull images. Banger flattens every pull into a self-contained ext4, so the cache is purely a re-pull avoidance — losing it only costs network round-trips on the next pull of the same image. Reclaim disk with:

banger image cache prune --dry-run   # report size only
banger image cache prune             # remove every cached blob

Run with the daemon idle; an in-flight pull racing against prune may fail and need a retry.

Tech debt

  • Auth. When we add private-registry support, the natural path is authn.DefaultKeychain, which honours ~/.docker/config.json and the standard credential helpers.
  • Non-systemd rootfses. The guest agents assume systemd. Adding openrc / s6 / busybox-init variants means keeping parallel unit trees keyed on /etc/os-release.

Trust model

image pull (OCI path) delegates trust to the registry the user selected. go-containerregistry verifies layer digests against the manifest during download, so a tampered mirror can't ship modified layers without breaking the sha256 chain. Banger does not verify OCI image signatures (cosign/sigstore) — users who care should verify references out-of-band.