Second phase of splitting the daemon god-struct. ImageService now owns
all image + kernel registry operations: register/promote/delete/pull
for images (bundle + OCI paths), the six kernel commands, and the
shared SSH-key/work-seed injection helpers. imageOpsMu (the
publication-window lock) lives on the service; so do the three OCI
pull test seams pullAndFlatten / finalizePulledRootfs / bundleFetch.
The four files images.go, images_pull.go, image_seed.go, kernels.go
flipped their receivers from *Daemon to *ImageService.
FindImage moved with the service. Daemon keeps a thin FindImage
forwarder so callers reading the dispatch code see the obvious
facade and tests that pre-date the split still compile.
flattenNestedWorkHome — called from image_seed.go, vm_authsync.go,
and vm_disk.go across future service boundaries — became a
package-level helper taking a CommandRunner explicitly. Daemon keeps
a deprecated forwarder for now; the other services will use the
package form.
Lazy-init helper imageSvc() on Daemon mirrors hostNet() from
Phase 1, so test literals like &Daemon{store: db, runner: r, ...}
that don't spell out an ImageService still get a working one.
Tests that override the image test seams (autopull_test,
concurrency_test, images_pull_test, images_pull_bundle_test) now
assign d.img = &ImageService{...seams...}; the two-statement pattern
matches what Phase 1 established for HostNetwork.
Dispatch in daemon.go is cleaner now: every image/kernel RPC handler
is a single-liner forwarding to d.imageSvc().*. Phase 5 will do the
same for VM lifecycle.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Before: createVMMu was held across the whole of CreateVM — including
image resolution (which could fire a full auto-pull) and startVMLocked
(boot of multiple seconds). imageOpsMu was held across the whole of
PullImage/RegisterImage/PromoteImage/DeleteImage, so any slow OCI pull,
bundle download, or file copy blocked every other image mutation and
every other VM create that needed to auto-pull. The async create API
bought nothing if all creates serialised on the same mutex.
CreateVM is now three phases:
1. Validate + resolve image (possibly auto-pulling). No global lock.
2. reserveVM: take createVMMu only long enough to re-check the name
is free, allocate the next guest IP, and UpsertVM the "created"
row. Milliseconds.
3. startVMLocked: run the full boot flow under the per-VM lock only.
Parallel creates of different VMs now overlap on image resolution +
boot; they contend only across the reservation claim.
For the image surface a new publishImage helper isolates the commit
atom (recheck name free, atomic rename stagingDir→finalDir, UpsertImage)
under imageOpsMu. pullFromBundle + pullFromOCI do their network fetch
+ ext4 build + ownership fixup + agent injection outside the lock;
Register moves validation + kernel resolution outside; Promote moves
file copy + SSH-key seeding outside; Delete keeps a brief lock over
the lookup + reference check + store delete and does file cleanup
unlocked.
Two concurrency tests assert the new behaviour:
- TestPullImageDoesNotSerialiseOnDifferentNames fails the old code
(second pull blocks on imageOpsMu and never reaches the body).
- TestPullImageRejectsNameClashAtPublish confirms the publish-window
recheck is what enforces name uniqueness now that the body runs
unlocked — exactly one winner.
ARCHITECTURE.md updated to describe the new scope explicitly instead
of calling the locks "narrow".
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
The `image build` flow spun up a transient Firecracker VM, SSHed in,
and ran a large bash provisioning script to derive a new managed
image from an existing one. It overlapped heavily with the golden-
image Dockerfile flow (same mise/docker/tmux/opencode install logic
duplicated in Go as `imagemgr.BuildProvisionScript`) and had far more
machinery: async op state, RPC begin/status/cancel, webui form +
operation page, preflight checks, API types, tests. For custom
images, writing a Dockerfile is simpler and more reproducible.
Removed end-to-end:
- CLI `image build` subcommand + `absolutizeImageBuildPaths`.
- Daemon: BuildImage method, imagebuild.go (transient-VM orchestration),
image_build_ops.go (async begin/status/cancel), imagemgr/build.go
(the 247-line provisioning script generator and all its append*
helpers), validateImageBuildPrereqs + addImageBuildPrereqs.
- RPC dispatches for image.build / .begin / .status / .cancel.
- opstate registry `imageBuildOps`, daemon seam `imageBuild`,
background pruner call.
- API types: ImageBuildParams, ImageBuildOperation, ImageBuildBeginResult,
ImageBuildStatusParams, ImageBuildStatusResult; model type
ImageBuildRequest.
- Web UI: Backend interface methods, handlers, form, routes, template
branches (images.html build form, operation.html build branch,
dashboard.html Build button).
- Tests that directly exercised BuildImage.
Doctor polish (task C):
- Drop the "image build" preflight section entirely (its raison d'être
is gone).
- Default-image check now accepts "not local but in imagecat" as OK:
vm create auto-pulls on first use. Only flag when the image is
neither locally registered nor in the catalog.
Net: 24 files touched, 1,373 lines deleted, 25 added.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
The golden-image Dockerfile + catalog pipeline replaces the entire
manual rootfs-build stack. With that shipped, the per-distro shell
flows are dead code.
Removed:
- scripts/customize.sh, scripts/interactive.sh, scripts/verify.sh
- scripts/make-rootfs{,-void,-alpine}.sh
- scripts/register-{void,alpine}-image.sh
- scripts/make-{void,alpine}-kernel.sh
- internal/imagepreset/ (only consumer was `banger internal packages`,
which fed customize.sh)
- examples/{void,alpine}.config.toml
- Makefile targets: rootfs, rootfs-void, rootfs-alpine, void-kernel,
alpine-kernel, void-register, alpine-register, void-vm, alpine-vm,
verify-void, verify-alpine, plus the ALPINE_RELEASE / *_IMAGE_NAME
/ *_VM_NAME variables
The void-6.12 kernel catalog entry is also gone — golden image pairs
with generic-6.12 and nothing else in the catalog depended on it.
Consolidated: imagemgr now holds the small DebianBasePackages list +
package-hash helper inline, so the `image build --from-image` flow
(still supported) no longer pulls from a separate imagepreset package.
Net: 3,815 lines deleted, 59 added. No runtime functionality removed
beyond the `banger internal packages` subcommand (hidden, used only
by the deleted customize.sh).
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
One-command sandbox: `banger vm run` on a fresh host now Just Works.
No prior `banger image pull` or `banger kernel pull` needed.
Changes:
- Default `default_image_name` flips from "default" to "debian-bookworm"
so the golden image is the implicit target when `--image` is omitted.
- `CreateVM` resolves the image via a new `findOrAutoPullImage`: try
the local store first, and on miss fall back to the embedded imagecat
catalog + auto-pull. Emits a vm-create progress stage so the user
sees "pulling from image catalog" in the create output.
- `resolveKernelInputs` gains context + the same pattern via
`readOrAutoPullKernel`: try the local kernelcat, and on miss look up
the embedded kernelcat and auto-pull. Fires whenever a bundle's
manifest references a kernel the user hasn't pulled yet, not just
during image pull — any CreateVM with an image that needs a kernel
not yet local will resolve it.
- `--image` help text updated on both `vm run` and `vm create`.
Six tests cover local-hit-no-pull, auto-pull-on-miss, not-in-catalog
error propagation, and a non-ENOENT kernel read error does NOT trigger
a misleading "not in catalog" claim.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
(d *Daemon).PullImage downloads an OCI image, flattens it into an
ext4 rootfs, and registers the result as a managed banger image.
Flow (internal/daemon/images_pull.go):
1. Parse + validate the OCI ref via go-containerregistry/name.
2. Derive a friendly default name from the ref ("debian-bookworm")
when --name is omitted.
3. Reject if an image with that name already exists.
4. Resolve kernel info via the new shared resolveKernelInputs
helper (refactored out of RegisterImage); ValidateKernelPaths
checks the kernel triple alone.
5. Acquire imageOpsMu, generate a fresh image id, and stage at
<ImagesDir>/<id>.staging.
6. imagepull.Pull → cache layers under OCICacheDir;
imagepull.Flatten → temp rootfs tree under os.TempDir (so the
state filesystem doesn't temporarily double in size).
7. Default size: max(treeSize × 1.25, 1 GiB); --size override
accepted.
8. imagepull.BuildExt4 produces the rootfs.ext4 in the staging dir.
9. imagemgr.StageBootArtifacts stages the kernel/initrd/modules
into the same dir (reused unchanged).
10. Atomic os.Rename(staging, finalDir) publishes the artifact dir.
11. Persist model.Image with Managed=true. Failure at any step
removes the staging dir; failure post-rename removes finalDir.
The pullAndFlatten field on Daemon is the test seam: tests stub it
to write a fixture tree into destDir and skip the real registry.
Refactor: extracted the "kernel-ref vs direct paths" resolution
out of RegisterImage into d.resolveKernelInputs so PullImage and
RegisterImage share one source of truth for that policy. Split
ValidateRegisterPaths into a kernel-only ValidateKernelPaths so
PullImage (which produces the rootfs itself) can validate just
the kernel triple without the rootfs check.
API: ImagePullParams { Ref, Name, KernelPath, InitrdPath,
ModulesDir, KernelRef, SizeBytes }. RPC dispatch case image.pull
mirrors image.register.
Tests cover: happy-path producing a managed image with all four
artifacts present + staging cleaned up, name-collision rejection,
missing-kernel rejection, and staging cleanup on a failed pull.
defaultImageNameFromRef handles tag/digest/no-suffix cases.
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
`banger image register --kernel-ref <name>` now substitutes for the
--kernel/--initrd/--modules triple. The daemon looks the name up via
kernelcat.ReadLocal under d.layout.KernelsDir, populates the three
paths from the resolved entry, then continues through the existing
validate/persist flow unchanged.
Passing both --kernel-ref and any of --kernel/--initrd/--modules is
rejected — at the CLI layer (before starting the daemon) and
defensively at the RPC layer. A missing catalog entry produces a clear
"run 'banger kernel list'" message.
Once registered, the image stores the resolved absolute paths, so
deleting the catalog entry later does not invalidate already-registered
images — managed image build still copies the kernel into its artifact
dir per imagemgr.StageBootArtifacts.
Tests cover: resolution success (absolute KernelPath populated from
catalog), mutual-exclusion rejection, and missing-entry error.
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
Moves the stateless helpers of the image subsystem into
internal/daemon/imagemgr:
paths.go — path validators (ValidateRegisterPaths,
ValidatePromotePaths), artifact staging (StageBootArtifacts,
StageOptionalArtifactPath), metadata (BuildMetadataPackages,
WritePackagesMetadata).
build.go — ResizeRootfs, WriteBuildLog, and the full guest
provisioning script generator (BuildProvisionScript, BuildModulesCommand
and all private script-append helpers) along with the mise/tmux/opencode
version constants.
The orchestrator methods (BuildImage, RegisterImage, PromoteImage,
DeleteImage, runImageBuildNative) stay on *Daemon: they still touch
d.store, d.imageOpsMu, d.beginOperation, capability hooks, and
fcproc-wrapped Daemon helpers — extracting them needs prerequisite
phases (operation protocol, workdisk helpers, tap pool). This commit is
strictly the pure-helper extraction that can land cleanly today.
imagebuild.go shrinks from 453 -> 225 LOC (half gone). images.go shrinks
from 450 -> 374 LOC. imagebuild_test.go updated to call the exported
imagemgr.BuildProvisionScript. Zero behavior change; all tests green.
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
Daemon no longer owns a coarse mu shared across unrelated concerns.
Each subsystem now carries its own state and lock:
- tapPool: entries, next, and mu move onto a new tapPool struct.
- sessionRegistry: sessionControllers + its mutex move off Daemon.
- opRegistry[T asyncOp]: generic registry collapses the two ad-hoc
vm-create and image-build operation maps (and their mutexes) into one
shared type; the Begin/Status/Cancel/Prune methods simplify.
- vmLockSet: the sync.Map of per-VM mutexes moves into its own type;
lockVMID forwards.
- Daemon.mu splits into imageOpsMu (image-registry mutations) and
createVMMu (CreateVM serialisation) so image ops and VM creates no
longer block each other.
Lock ordering collapses to vmLocks[id] -> {createVMMu, imageOpsMu} ->
subsystem-local leaves. doc.go and ARCHITECTURE.md updated.
No behavior change; tests green.
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
Hard-cut banger away from source-checkout runtime bundles as an implicit source of\nimage and host defaults. Managed images now own their full boot set,\nimage build starts from an existing registered image, and daemon startup\nno longer synthesizes a default image from host paths.\n\nResolve Firecracker from PATH or firecracker_bin, make SSH keys config-owned\nwith an auto-managed XDG default, replace the external name generator and\npackage manifests with Go code, and keep the vsock helper as a companion\nbinary instead of a user-managed runtime asset.\n\nUpdate the manual scripts, web/CLI forms, config surface, and docs around\nthe new build/manual flow and explicit image registration semantics.\n\nValidation: GOCACHE=/tmp/banger-gocache go test ./..., bash -n scripts/*.sh,\nand make build.
Add a localhost-only web console so VM and image management no longer depends on the CLI for every inspection and lifecycle action.
Wire bangerd up to a configurable web listener, expose dashboard and async image-build state through the daemon, and serve CSRF-protected HTML pages with host-path picking, VM/image detail views, logs, ports, and progress polling for long-running operations.
Keep the browser path aligned with the existing sudo and host-owned artifact model: surface sudo readiness, print the web URL in daemon status, and document the new workflow. Polish the UI with resource usage cards, clearer clickable affordances, cancel paths, confirmation prompts, image-name links, and HTTP port links.
Validation: GOCACHE=/tmp/banger-gocache go test ./...
Stop relying on ad hoc rootfs handling by adding image promotion, managed work-seed fingerprint metadata, and lazy self-healing for older managed images after the first create.
Rebuild guest images with baked SSH access, a guest NIC bootstrap, and default opencode services, and add the staged Void kernel/initramfs/modules workflow so void-exp uses a matching Void boot stack.
Replace the opaque blocking vm.create RPC with a begin/status flow that prints live stages in the CLI while still waiting for vsock health and opencode on guest port 4096.
Validate with GOCACHE=/tmp/banger-gocache go test ./... and live void-exp create/delete smoke runs.
Make iterating on a Firecracker-friendly Void guest practical without replacing the Debian default image path.
Add local Void rootfs build/register/verify plumbing, a language-agnostic dev package baseline, and guest SSH/work-disk hardening so new images use the runtime bundle key, keep a normal root bash environment, and repair stale nested /root layouts on restart.
Replace the guest PING/PONG responder with an HTTP /healthz agent over vsock, rename the runtime bundle and config surface from ping helper to agent while still accepting the legacy keys, and route the post-SSH reminder through the new vm.health path.
Validated with GOCACHE=/tmp/banger-gocache go test ./..., make build, bash -n customize.sh make-rootfs-void.sh, and git diff --check.
Beat VM create wall time without changing VM semantics.
Generate a work-seed ext4 sidecar during image builds and rootfs rebuilds, then clone and resize that seed for each new VM instead of rebuilding /root from scratch. Plumb the new seed artifact through config, runtime metadata, store state, runtime-bundle defaults, doctor checks, and default-image reconciliation so older images still fall back cleanly.
Add a daemon TAP pool to keep idle bridge-attached devices warm, expose stage timing in lifecycle logs, add a create/SSH benchmark script plus Make target, and teach verify.sh that tap-pool-* devices are reusable capacity rather than cleanup leaks.
Validated with go test ./..., make build, ./verify.sh, and make bench-create ARGS="--runs 2".
Reduce the control plane's dependency on helper scripts while keeping the hard Linux integration points in the approved shell-out layer.
Replace the bash-driven image build path with a native Go builder that clones and optionally resizes the rootfs, boots a temporary Firecracker VM, provisions the guest over SSH, installs packages and modules, and preserves the package-manifest sidecar.
Also replace a few small convenience shell-outs with Go helpers: read process stats from /proc, use os.Truncate for ext4 image growth, add file-clone and normalized-line helpers, drop the sh -c work-disk flattening path, and launch Firecracker via a direct sudo command.
Add tests for the new SSH/archive and system helpers, plus a policy test that keeps os/exec imports confined to cli/firecracker/system. Update the docs to describe customize.sh as a manual helper rather than the daemon's image-build backend.
Validated with go mod tidy, go test ./..., and make build.
Serve daemon-managed .vm names directly from bangerd on 127.0.0.1:42069 instead of shelling out to mapdns. This keeps DNS state tied to VM lifecycle and lets the daemon rebuild records from running VMs after startup or reconcile.
Add a small in-process authoritative DNS server, register and remove records from the VM start/stop/delete paths, and show the listener in daemon status. Remove the mapdns config and preflight surface, stop helper-flow DNS publishing in customize.sh and interactive.sh, drop dns.sh from the runtime bundle, and update docs/tests for the new local-resolver integration model.
Validated with GOCACHE=/tmp/banger-gocache go test ./..., GOCACHE=/tmp/banger-gocache make build, and bash -n customize.sh interactive.sh.
VM start, image build, and network/setup failures were hard to diagnose because bangerd emitted almost no lifecycle logs and the Firecracker SDK logger was discarded. This adds a daemon-wide JSON logger with configurable log level so failures leave breadcrumbs instead of only side effects.
Log the main daemon and VM lifecycle stages, preserve raw Firecracker and image-build helper output in dedicated files, and include those log paths in daemon status and returned errors. Bridge SDK logrus output into the daemon logger at debug level so low-level Firecracker diagnostics are available without making normal info logs unreadable.
Validation: go test ./... and make build. Left unrelated worktree changes out of this commit, including internal/api/types.go, the deleted shell scripts, and my-rootfs.ext4.
Stop assuming one workstation layout for runtime artifacts, mapdns, and host tooling. The daemon and shell helpers now use portable mapdns configuration, and runtime bundles can carry bundle.json metadata for their default kernel, initrd, modules, rootfs, and helper paths.
Load bundle metadata through config with a legacy layout fallback, thread mapdns_bin/mapdns_data_file through the Go and shell paths, and add command-scoped preflight checks for VM start, NAT, image build, work-disk resize, and SSH so missing tools or artifacts fail with actionable errors.
Update the runtime-bundle manifest, docs, and tests to match the new model. Verified with go test ./..., make build, and bash -n customize.sh interactive.sh dns.sh make-rootfs.sh verify.sh.
Stop treating Firecracker, kernels, modules, and guest images as tracked source files. Source checkouts now resolve runtime assets from ./runtime, while installed binaries keep using ../lib/banger.
Add a small runtimebundle helper plus runtime-bundle.toml so make can bootstrap, package, and install a runtime bundle with checksum validation. Update the shell helpers and daemon path hints to fail clearly when the bundle is missing instead of assuming repo-root artifacts.
This removes the tracked runtime blobs from HEAD in favor of an ignored local runtime/ tree. Verified with go test ./..., make build, bash -n on the shell helpers, make -n install, and a temporary package/fetch smoke test. The manifest URL/SHA still need a published bundle before fresh clones can bootstrap, and history rewrite remains a separate rollout step.
Fix the misleading make install path where banger and bangerd still depended on a repo checkout for Firecracker, guest artifacts, image builds, and the SSH key.
Replace repo-root inference with an explicit runtime bundle model: resolve a runtime_dir from env/config/install layout, derive concrete artifact paths from it, and update the daemon, CLI, and image-build flow to use those paths. Keep repo_root only as an explicit compatibility alias instead of auto-detecting it.
Teach customize.sh to run from a read-only bundled runtime tree while writing transient state under XDG/BANGER_STATE_DIR, and make make install copy the runtime assets into PREFIX/lib/banger so installed binaries stay usable outside the repo.
Validate with go test ./..., make build, bash -n customize.sh, and make install DESTDIR=/tmp/banger-install PREFIX=/usr. An out-of-repo installed-binary smoke test was attempted, but this sandbox blocked bangerd from binding its Unix socket (setsockopt: operation not permitted).
Replace the shell-only user workflow with `banger` and `bangerd`: Cobra commands, XDG/SQLite-backed state, managed VM and image lifecycle, and a Bubble Tea TUI for browsing and operating VMs.\n\nKeep Firecracker orchestration behind the daemon so VM specs become persistent objects, and add repo entrypoints for building, installing, and documenting the new flow while still delegating rootfs customization to the existing shell tooling.\n\nHarden the control plane around real usage by reclaiming Firecracker API sockets for the user, restarting stale daemons after rebuilds, and returning the correct `vm.create` payload so the CLI and TUI creation flow work reliably.\n\nValidation: `go test ./...`, `make build`, and a host-side smoke test with `./banger vm create --name codex-smoke`.
