diff --git a/.gitignore b/.gitignore index a446740..b5c9f77 100644 --- a/.gitignore +++ b/.gitignore @@ -16,3 +16,4 @@ wtf/*.deb *.pem *.key id_rsa +.env diff --git a/AGENTS.md b/AGENTS.md index 1a5f801..1ab1f7f 100644 --- a/AGENTS.md +++ b/AGENTS.md @@ -20,6 +20,7 @@ Always run `make build` before commit. - `./build/bin/banger image register ...` registers an unmanaged host-side image stack. - `./build/bin/banger image promote ` copies an unmanaged image into daemon-owned managed artifacts. - `make void-kernel`, `make rootfs-void`, and `make void-register` drive the experimental Void flow under `./build/manual`. +- `scripts/publish-kernel.sh ` packages a locally-imported kernel and uploads it to the catalog; see `docs/kernel-catalog.md`. ## Image Model diff --git a/README.md b/README.md index f1960fc..206b300 100644 --- a/README.md +++ b/README.md @@ -88,6 +88,20 @@ Register an existing host-side image stack: --modules /abs/path/modules ``` +Or pull a pre-built kernel from the catalog and reference it by name: + +```bash +./build/bin/banger kernel list --available +./build/bin/banger kernel pull void-6.12 +./build/bin/banger image register \ + --name base \ + --rootfs /abs/path/rootfs.ext4 \ + --kernel-ref void-6.12 +``` + +See [`docs/kernel-catalog.md`](docs/kernel-catalog.md) for catalog +maintenance. + Build a managed image from an existing registered image: ```bash diff --git a/docs/kernel-catalog.md b/docs/kernel-catalog.md new file mode 100644 index 0000000..c616e8d --- /dev/null +++ b/docs/kernel-catalog.md @@ -0,0 +1,117 @@ +# Kernel catalog + +The kernel catalog ships pre-built Firecracker-ready kernel bundles so users +don't have to compile anything. The catalog is embedded into the banger +binary and updated each release. + +End-user flow: + +```bash +banger kernel list --available # browse the catalog +banger kernel pull void-6.12 # download a bundle (no sudo, no make) +banger image register --name void --rootfs … --kernel-ref void-6.12 +``` + +## Architecture + +Two parts: + +1. **`internal/kernelcat/catalog.json`** — a JSON manifest embedded into the + banger binary via `go:embed`. Each entry carries a name, distro, arch, + kernel version, tarball URL, and tarball SHA256. Updating the catalog + means editing this file in the repo and rebuilding banger. + +2. **Tarballs at `https://kernels.thaloco.com/`** — Cloudflare R2 bucket + `banger-kernels`, fronted by a public custom domain. Each tarball is + `-.tar.zst` and contains `vmlinux`, optional `initrd.img`, + and an optional `modules/` tree at the archive root. + +The `banger kernel pull` flow streams the tarball, verifies its SHA256 +against the embedded catalog entry, decompresses it (zstd), extracts it +into `~/.local/state/banger/kernels//`, and writes a manifest. Path +traversal entries and unsafe symlinks are rejected. + +## Adding or updating an entry + +The repo has no CI for kernel publishing yet. Catalog updates are manual +and infrequent (kernel version bumps every few weeks at most). + +```bash +# 1. Build the kernel locally with the existing helper. +make void-kernel # or: make alpine-kernel + +# 2. Import it into the local catalog so the canonical layout exists. +banger kernel import void-6.12 \ + --from build/manual/void-kernel \ + --distro void \ + --arch x86_64 + +# 3. Package, upload, patch catalog.json. +scripts/publish-kernel.sh void-6.12 \ + --description "Void Linux 6.12 kernel for Firecracker microVMs" + +# 4. Review and commit the catalog change. +git diff -- internal/kernelcat/catalog.json +git add internal/kernelcat/catalog.json +git commit -m 'kernel catalog: add/update void-6.12' + +# 5. Rebuild so the new catalog is embedded. +make build +``` + +`scripts/publish-kernel.sh` reads the locally-imported entry under +`~/.local/state/banger/kernels//`, builds a tar+zstd archive, uploads +it to R2 via `rclone`, HEAD-checks the public URL, and patches +`internal/kernelcat/catalog.json` with the new URL, SHA256, and size. + +Environment overrides if the defaults need to change: +`RCLONE_REMOTE`, `RCLONE_BUCKET`, `BASE_URL`, `BANGER_KERNELS_DIR`. + +## Removing an entry + +1. Delete the line from `internal/kernelcat/catalog.json` and commit. +2. Delete the tarball from R2: `rclone delete r2:banger-kernels/-.tar.zst`. +3. Rebuild banger. + +Already-pulled local copies on user machines are not invalidated — they +keep working until the user runs `banger kernel rm `. That's +intentional: pulling is idempotent, removing should not break anyone in +the middle of a workflow. + +## Versioning conventions + +- **Entry names**: `-` (e.g. `void-6.12`, + `alpine-3.23`). The major.minor is the kernel line, not the distro + release. Patch-level bumps reuse the entry name and replace the + tarball; minor bumps create a new entry (`void-6.13`). +- **Architecture**: only `x86_64` is published today. The `arch` field in + the catalog schema is additive — adding `arm64` later is a config + change, not a schema change. +- **Tarball layout**: contents at the archive root (no top-level + versioned directory). `vmlinux` is required; `initrd.img` and + `modules/` are optional. Symlinks inside `modules/` are allowed but + must resolve within the archive. + +## Trust model + +The embedded `catalog.json` carries the SHA256 of each tarball. `banger +kernel pull` rejects any download whose hash doesn't match. This protects +against transport corruption and against an attacker swapping a tarball +on R2 without also pushing a banger release. + +It does **not** protect against a compromise of the banger source repo +itself — an attacker who can land a commit can change both the catalog +SHA256 and the tarball. GPG/sigstore signing is deferred until banger is +public and the threat model justifies the operational overhead. + +## Hosting + +Tarballs live in Cloudflare R2 (bucket `banger-kernels`), served at the +custom domain `kernels.thaloco.com`. The bucket is publicly readable; +writes require the `banger-kernels-publish` API token (kept locally, +never committed). R2's free tier covers the expected traffic comfortably +(zero egress fees, generous storage). + +If hosting ever moves, catalog entries can be migrated by reuploading the +tarballs and editing the URLs in `catalog.json` — no other code changes +required. diff --git a/scripts/publish-kernel.sh b/scripts/publish-kernel.sh new file mode 100755 index 0000000..c627936 --- /dev/null +++ b/scripts/publish-kernel.sh @@ -0,0 +1,139 @@ +#!/usr/bin/env bash +# publish-kernel.sh +# +# Package an entry from the local banger kernel catalog as a tar.zst, +# upload it to the public R2 bucket, and patch internal/kernelcat/catalog.json +# with the resulting URL + sha256 + size. Run after `banger kernel import`. +# +# Usage: +# scripts/publish-kernel.sh [--description "..."] +# +# Environment overrides: +# RCLONE_REMOTE rclone remote to upload through (default: r2) +# RCLONE_BUCKET R2 bucket name (default: banger-kernels) +# BASE_URL public URL prefix for the bucket (default: https://kernels.thaloco.com) +# BANGER_KERNELS_DIR local catalog directory (default: ~/.local/state/banger/kernels) + +set -euo pipefail + +log() { printf '[publish-kernel] %s\n' "$*" >&2; } +die() { log "$*"; exit 1; } + +usage() { + cat < [--description ""] + +Reads the locally-imported kernel at \$BANGER_KERNELS_DIR//, packages +it as -.tar.zst, uploads to R2, and updates +internal/kernelcat/catalog.json. + +Run \`banger kernel import --from --distro --arch \` +first. +EOF +} + +RCLONE_REMOTE="${RCLONE_REMOTE:-r2}" +RCLONE_BUCKET="${RCLONE_BUCKET:-banger-kernels}" +BASE_URL="${BASE_URL:-https://kernels.thaloco.com}" +BANGER_KERNELS_DIR="${BANGER_KERNELS_DIR:-$HOME/.local/state/banger/kernels}" + +SCRIPT_DIR="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)" +REPO_ROOT="$(cd "$SCRIPT_DIR/.." && pwd)" +CATALOG_FILE="$REPO_ROOT/internal/kernelcat/catalog.json" + +NAME="" +DESCRIPTION="" +while [[ $# -gt 0 ]]; do + case "$1" in + -d|--description) DESCRIPTION="${2:-}"; shift 2;; + -h|--help) usage; exit 0;; + --) shift; break;; + -*) die "unknown flag: $1";; + *) + if [[ -z "$NAME" ]]; then + NAME="$1"; shift + else + die "unexpected positional arg: $1" + fi + ;; + esac +done +[[ -n "$NAME" ]] || { usage; exit 1; } + +for tool in jq rclone tar zstd sha256sum stat curl; do + command -v "$tool" >/dev/null 2>&1 || die "missing required tool: $tool" +done +[[ -f "$CATALOG_FILE" ]] || die "catalog file not found: $CATALOG_FILE" + +SRC="$BANGER_KERNELS_DIR/$NAME" +[[ -d "$SRC" ]] || die "$SRC does not exist; run 'banger kernel import $NAME --from ' first" +[[ -f "$SRC/vmlinux" ]] || die "$SRC/vmlinux missing" +[[ -f "$SRC/manifest.json" ]] || die "$SRC/manifest.json missing" + +DISTRO="$(jq -r '.distro // ""' "$SRC/manifest.json")" +ARCH="$(jq -r '.arch // ""' "$SRC/manifest.json")" +KERNEL_VERSION="$(jq -r '.kernel_version // ""' "$SRC/manifest.json")" +[[ -n "$ARCH" ]] || ARCH="x86_64" + +STAGE="$(mktemp -d)" +trap 'rm -rf "$STAGE"' EXIT + +TARBALL_NAME="${NAME}-${ARCH}.tar.zst" +TARBALL="$STAGE/$TARBALL_NAME" + +INCLUDES=(vmlinux) +[[ -f "$SRC/initrd.img" ]] && INCLUDES+=(initrd.img) +[[ -d "$SRC/modules" ]] && INCLUDES+=(modules) + +log "packaging ${INCLUDES[*]} from $SRC" +( cd "$SRC" && tar -cf - "${INCLUDES[@]}" ) | zstd -19 --long -T0 -q -o "$TARBALL" + +SHA256="$(sha256sum "$TARBALL" | awk '{print $1}')" +SIZE="$(stat -c '%s' "$TARBALL")" +HUMAN_SIZE="$(numfmt --to=iec --suffix=B "$SIZE" 2>/dev/null || echo "${SIZE}B")" +log "tarball $TARBALL_NAME: $HUMAN_SIZE, sha256 $SHA256" + +log "uploading to $RCLONE_REMOTE:$RCLONE_BUCKET/$TARBALL_NAME" +rclone copyto "$TARBALL" "$RCLONE_REMOTE:$RCLONE_BUCKET/$TARBALL_NAME" + +URL="$BASE_URL/$TARBALL_NAME" +log "verifying $URL is reachable" +HEAD_STATUS="$(curl -fsSI -o /dev/null -w '%{http_code}' "$URL" || true)" +if [[ "$HEAD_STATUS" != "200" ]]; then + die "uploaded tarball is not publicly reachable at $URL (HTTP $HEAD_STATUS); check bucket public-access config" +fi + +log "patching $CATALOG_FILE" +NEW_ENTRY="$(jq -n \ + --arg name "$NAME" \ + --arg distro "$DISTRO" \ + --arg arch "$ARCH" \ + --arg kver "$KERNEL_VERSION" \ + --arg url "$URL" \ + --arg sha "$SHA256" \ + --argjson size "$SIZE" \ + --arg desc "$DESCRIPTION" \ + '{ + name: $name, + distro: $distro, + arch: $arch, + kernel_version: $kver, + tarball_url: $url, + tarball_sha256: $sha, + size_bytes: $size, + description: $desc + } | with_entries(select(.value != null and .value != ""))')" + +CATALOG_TMP="$(mktemp)" +jq --arg name "$NAME" --argjson new "$NEW_ENTRY" ' + .version = (.version // 1) + | .entries = (((.entries // []) | map(select(.name != $name))) + [$new]) + | .entries |= sort_by(.name) +' "$CATALOG_FILE" > "$CATALOG_TMP" +mv "$CATALOG_TMP" "$CATALOG_FILE" + +log "done" +log "next steps:" +log " git diff -- $CATALOG_FILE" +log " git add $CATALOG_FILE && git commit -m 'kernel catalog: add/update $NAME'" +log " make build # rebuild banger so the new catalog is embedded"