daemon: shrink createVMMu + imageOpsMu to reservation/publication windows

Before: createVMMu was held across the whole of CreateVM — including
image resolution (which could fire a full auto-pull) and startVMLocked
(boot of multiple seconds). imageOpsMu was held across the whole of
PullImage/RegisterImage/PromoteImage/DeleteImage, so any slow OCI pull,
bundle download, or file copy blocked every other image mutation and
every other VM create that needed to auto-pull. The async create API
bought nothing if all creates serialised on the same mutex.

CreateVM is now three phases:

 1. Validate + resolve image (possibly auto-pulling). No global lock.
 2. reserveVM: take createVMMu only long enough to re-check the name
    is free, allocate the next guest IP, and UpsertVM the "created"
    row. Milliseconds.
 3. startVMLocked: run the full boot flow under the per-VM lock only.

Parallel creates of different VMs now overlap on image resolution +
boot; they contend only across the reservation claim.

For the image surface a new publishImage helper isolates the commit
atom (recheck name free, atomic rename stagingDir→finalDir, UpsertImage)
under imageOpsMu. pullFromBundle + pullFromOCI do their network fetch
+ ext4 build + ownership fixup + agent injection outside the lock;
Register moves validation + kernel resolution outside; Promote moves
file copy + SSH-key seeding outside; Delete keeps a brief lock over
the lookup + reference check + store delete and does file cleanup
unlocked.

Two concurrency tests assert the new behaviour:
 - TestPullImageDoesNotSerialiseOnDifferentNames fails the old code
   (second pull blocks on imageOpsMu and never reaches the body).
 - TestPullImageRejectsNameClashAtPublish confirms the publish-window
   recheck is what enforces name uniqueness now that the body runs
   unlocked — exactly one winner.

ARCHITECTURE.md updated to describe the new scope explicitly instead
of calling the locks "narrow".

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
This commit is contained in:
Thales Maciel 2026-04-20 13:44:22 -03:00
parent afe91e805a
commit 99d0811097
No known key found for this signature in database
GPG key ID: 33112E6833C34679
5 changed files with 390 additions and 95 deletions

View file

@ -31,10 +31,23 @@ owning types:
reconstruct the cache and verify processes against `/proc` via
pgrep. Nothing in the durable `vms` SQLite row describes transient
kernel state. See `internal/daemon/vm_handles.go`.
- `createVMMu sync.Mutex` — serialises `CreateVM` (guards name uniqueness
+ guest IP allocation window).
- `imageOpsMu sync.Mutex` — serialises image-registry mutations
(`PullImage`, `RegisterImage`, `PromoteImage`, `DeleteImage`).
- `createVMMu sync.Mutex` — narrow **reservation** mutex. `CreateVM`
resolves the image (possibly auto-pulling, which self-locks on
`imageOpsMu`) and parses sizing flags outside this lock, then holds
`createVMMu` only to re-check that the requested VM name is still
free, allocate the next guest IP, and insert the initial "created"
row. The subsequent boot flow runs under the per-VM lock only.
Parallel `vm create` calls therefore overlap on image resolution and
boot; they contend only across the millisecond-scale name+IP claim.
- `imageOpsMu sync.Mutex` — narrow **publication** mutex. `PullImage`
(both bundle and OCI paths), `RegisterImage`, `PromoteImage`, and
`DeleteImage` do their slow work (network fetch, ext4 build,
ownership fixup, file copy, SSH-key seeding) without this lock and
acquire it only for the commit atom: recheck name free, atomic
rename of the staging dir to its final home, upsert the store row.
Two pulls for different images run fully in parallel; two pulls that
race to the same name are resolved at the recheck — the loser fails
fast and its staging dir is cleaned up.
- `createOps opstate.Registry[*vmCreateOperationState]` — in-flight VM
create operations; owns its own lock.
- `tapPool tapPool` — TAP interface pool; owns its own lock.
@ -93,8 +106,13 @@ Notes:
- `vmLocks[id]` is the outer lock for any operation scoped to a single VM.
Acquired via `withVMLockByID` / `withVMLockByRef`. The callback runs
under the lock — treat the whole function body as critical section.
- `createVMMu` and `imageOpsMu` are narrow: each guards one family of
mutations and is released before any blocking guest I/O.
- `createVMMu` is held only across the VM-name reservation + IP
allocation + initial UpsertVM. Image resolution and the full boot
flow happen outside it.
- `imageOpsMu` is held only across the publication atom (recheck name
+ atomic rename + UpsertImage, or the equivalent for Register /
Promote / Delete). Network fetch, ext4 build, and file copies run
unlocked.
- Holding a subsystem-local lock while calling into guest SSH is
discouraged; copy needed state out under the lock and release before
blocking I/O.