banger/internal/daemon/ARCHITECTURE.md

internal/daemon architecture

This document describes the current daemon package layout: the Daemon composition root, the four services it wires together, the subpackages that own stateless helpers, and the lock ordering every caller must respect.

Composition

Daemon is a thin composition root. It holds shared infrastructure (store, runner, logger, layout, config, listener) plus pointers to four focused services. RPC dispatch is a pure forwarder into those services; no lifecycle / image / workspace / networking behaviour lives on *Daemon itself.

Daemon
├── *HostNetwork      — bridge, tap pool, NAT, DNS, firecracker process,
│                       DM snapshots, vsock readiness
├── *ImageService     — register, promote, delete, pull (bundle + OCI),
│                       kernel catalog, managed-seed refresh
├── *WorkspaceService — workspace.prepare / workspace.export, auth-key
│                       + git-identity sync onto the work disk
└── *VMService        — VM lifecycle (create/start/stop/restart/kill/
                        delete/set), stats polling, ports query,
                        handle cache, per-VM lock set, create-op
                        registry, preflight validation

Each service owns its own state. Cross-service calls go through narrow consumer-defined seams:

• WorkspaceService does not hold a *VMService pointer. It takes function-typed deps (vmResolver, aliveChecker, withVMLockByRef, imageResolver, imageWorkSeed) so it sees exactly the operations it needs and nothing more. Those deps are captured as closures so construction-order cycles don't recur.
• VMService holds direct pointers to *HostNetwork, *ImageService, and *WorkspaceService. Orchestrating a VM start really does compose all three (bridge + tap + image resolution + work-disk sync), and declaring a function-typed interface for every call would balloon the surface for no win — services are unexported, so package-external code can never reach them.
• Capability hooks still take *Daemon as their receiver argument, but VMService calls into them through a capabilityHooks struct (function-typed bag) populated at construction. The service has no *Daemon pointer.

Lazy-init getters (d.hostNet(), d.imageSvc(), d.workspaceSvc(), d.vmSvc()) let existing test literals (&Daemon{store: db, runner: r}) keep working — the getter constructs the service from whatever is on the Daemon if nothing was pre-wired.

Service state

HostNetwork (host_network.go, nat.go, dns_routing.go, tap_pool.go, snapshot.go)

• tapPool — TAP interface pool, owns its own lock.
• vmDNS *vmdns.Server — in-process DNS server for .vm names.
• No direct VM-state access. Where an operation needs a VM's tap name (e.g. ensureNAT), the signature takes guestIP + tap string so the caller (VMService) resolves them first.

ImageService (image_service.go, images.go, images_pull.go, image_seed.go, kernels.go)

• imageOpsMu sync.Mutex — the publication-window lock. Held only across the recheck-name + atomic-rename + UpsertImage commit atom. Slow work (network fetch, ext4 build, SSH-key seeding) runs unlocked.
• Test seams pullAndFlatten, finalizePulledRootfs, bundleFetch are struct fields (not package globals), so tests inject per-instance fakes.

WorkspaceService (workspace_service.go, workspace.go, vm_authsync.go)

• workspaceLocks vmLockSet — per-VM mutex scoped to workspace.prepare / workspace.export. These ops acquire vmLocks[id] (on VMService) only long enough to validate VM state and snapshot the fields they need, then release it and acquire workspaceLocks[id] for the slow guest I/O phase. That keeps vm stop / delete / restart from queueing behind a running tar import.
• Test seams workspaceInspectRepo, workspaceImport are per-instance fields.

VMService (vm_service.go, vm_lifecycle.go, vm_create.go, vm_create_ops.go, vm_stats.go, vm_set.go, vm_disk.go, vm_handles.go, vm_authsync.go (via WorkspaceService), preflight.go, ports.go, vm.go)

• vmLocks vmLockSet — per-VM *sync.Mutex, one per VM ID. Held for the entire lifecycle op on that VM: start holds it across preflight, bridge setup, firecracker spawn, and post-boot wiring (seconds to tens of seconds). Two start/stop/delete/set calls against the same VM therefore serialise; calls against different VMs run independently.
• createVMMu sync.Mutex — narrow reservation mutex. CreateVM resolves the image (possibly auto-pulling, which self-locks on imageOpsMu) and parses sizing flags outside this lock, then holds createVMMu only to re-check that the requested VM name is still free, allocate the next guest IP, and insert the initial "created" row. The subsequent boot flow runs under the per-VM lock only.
• createOps opstate.Registry[*vmCreateOperationState] — in-flight async create operations; owns its own lock.
• handles *handleCache — in-memory map of per-VM transient kernel/ process handles (PID, tap device, loop devices, DM target). Each VM directory holds a small handles.json scratch file so the cache can be rebuilt at daemon startup.
• Test seams guestWaitForSSH, guestDial are per-instance fields.

Subpackages

Stateless helpers with no need for a service pointer live in subpackages. Each takes explicit dependencies (typically a system.Runner-compatible interface) and holds no global state beyond small test seams.

Subpackage	Purpose
internal/daemon/opstate	Generic Registry[T AsyncOp] for async-operation bookkeeping.
internal/daemon/dmsnap	Device-mapper COW snapshot create/cleanup/remove.
internal/daemon/fcproc	Firecracker process primitives (bridge, tap, binary, PID, kill, wait).
internal/daemon/imagemgr	Image subsystem pure helpers: validators, staging, build script gen.
internal/daemon/workspace	Workspace helpers: git inspection, copy prep, guest import script.

All subpackages are leaves — no intra-daemon subpackage imports another.

Lock ordering

Acquire in this order, release in reverse. Never acquire in the opposite direction.

VMService.vmLocks[id]  →  WorkspaceService.workspaceLocks[id]
                      →  {VMService.createVMMu, ImageService.imageOpsMu}
                      →  subsystem-local locks

vmLocks[id] and workspaceLocks[id] are NEVER held at the same time. workspace.prepare acquires vmLocks[id] just long enough to validate VM state, releases it, then acquires workspaceLocks[id] for the guest I/O phase. Regular lifecycle ops (start, stop, delete, set) do NOT do this split — they hold vmLocks[id] across the whole flow.

Subsystem-local locks (tapPool.mu, opstate.Registry mu, handleCache.mu) are leaves. They do not contend with each other.

Notes:

• vmLocks[id] is the outer lock for any operation scoped to a single VM. Acquired via VMService.withVMLockByID / withVMLockByRef. The callback runs under the lock — treat the whole function body as critical section.
• createVMMu is held only across the VM-name reservation + IP allocation + initial UpsertVM. Image resolution and the full boot flow happen outside it.
• imageOpsMu is held only across the publication atom (recheck name
  • atomic rename + UpsertImage, or the equivalent for Register / Promote / Delete). Network fetch, ext4 build, and file copies run unlocked.
• Holding a subsystem-local lock while calling into guest SSH is discouraged; copy needed state out under the lock and release before blocking I/O.

Reconcile and background work

Daemon.reconcile(ctx) is the orchestrator run at startup. It rehydrates the handle cache, reaps stale VMs, and republishes DNS records. Daemon.backgroundLoop() is the ticker fan-out — VMService.pollStats, VMService.stopStaleVMs, and VMService.pruneVMCreateOperations run on independent tickers.

External API

Only internal/cli imports this package. The surface is:

• daemon.Open(ctx) (*Daemon, error)
• (*Daemon).Serve(ctx) error
• (*Daemon).Close() error
• daemon.Doctor(...) — host diagnostics (no receiver).

All other methods live on the four services and are reached only through the RPC dispatch switch in daemon.go. They are free to move/rename during refactoring.