Streamline VM overlays and rootfs packages

Move the default guest package list into a repo manifest and record a hash beside built rootfs images so run/make-rootfs can warn when the docker-ready image is stale.

Switch the Firecracker launch path to a single sparse root overlay per VM instead of separate /home and /var disks, so many VMs can share the same base image while still installing packages under /var and working from /root.

Keep older images bootable by masking stale home.mount and var.mount units at boot, and scrub those obsolete fstab entries when customize.sh rebuilds an image. Verified with bash -n on the updated scripts; no live VM boot was run in this environment.
This commit is contained in:
Thales Maciel 2026-03-15 19:36:54 -03:00
parent 9191b7e370
commit 3cf33d1e0a
No known key found for this signature in database
GPG key ID: 33112E6833C34679
8 changed files with 206 additions and 204 deletions

View file

@ -15,6 +15,7 @@ Minimal Firecracker launcher.
- `wtf/root/lib/modules/6.8.0-94-generic/`: guest kernel modules - `wtf/root/lib/modules/6.8.0-94-generic/`: guest kernel modules
- `rootfs.ext4`: guest root filesystem (base image if present) - `rootfs.ext4`: guest root filesystem (base image if present)
- `rootfs-docker.ext4`: docker-ready guest rootfs (built via `make-rootfs.sh`) - `rootfs-docker.ext4`: docker-ready guest rootfs (built via `make-rootfs.sh`)
- `packages.apt`: apt packages baked into rebuilt guest images
- `id_ed25519`: SSH key for `root` - `id_ed25519`: SSH key for `root`
- `mapdns`: local DNS mapping CLI used to publish `<vm-name>.vm` → guest IP records - `mapdns`: local DNS mapping CLI used to publish `<vm-name>.vm` → guest IP records
@ -25,22 +26,21 @@ Minimal Firecracker launcher.
## Run Options ## Run Options
``` ```
./run.sh --name calm_otter --vcpu 4 --ram 2048 --home-size 6G ./run.sh --name calm-otter --vcpu 4 --ram 2048 --overlay-size 12G
``` ```
- `--name`: must be unique and match `[a-z0-9][a-z0-9-]{0,63}`. - `--name`: must be unique and match `[a-z0-9][a-z0-9-]{0,63}`.
- `--vcpu`: defaults to 2, max 16. - `--vcpu`: defaults to 2, max 16.
- `--ram`: MiB, defaults to 1024, max 32768. - `--ram`: MiB, defaults to 1024, max 32768.
- `--overlay-size`: writable dm-snapshot size for VM changes under `/`, including `/root` and `/var` (default: 8G).
- `--rootfs`: path to the rootfs image (default: `./rootfs-docker.ext4`). - `--rootfs`: path to the rootfs image (default: `./rootfs-docker.ext4`).
- `--kernel`: path to the kernel image (default: `./wtf/root/boot/vmlinux-6.8.0-94-generic`). - `--kernel`: path to the kernel image (default: `./wtf/root/boot/vmlinux-6.8.0-94-generic`).
- `--initrd`: path to the initrd image (default: `./wtf/root/boot/initrd.img-6.8.0-94-generic`). - `--initrd`: path to the initrd image (default: `./wtf/root/boot/initrd.img-6.8.0-94-generic`).
- `--home-size`: M/G suffixes supported (default: 2G).
- `--var-size`: M/G suffixes supported (default: 2G).
## Storage Layout ## Storage Layout
- `rootfs.ext4` is used as the read-only origin for a per-VM device-mapper snapshot mounted as `/`. - `rootfs.ext4` is used as the read-only origin for a per-VM device-mapper snapshot mounted as `/`.
- Each VM gets writable ext4 disks mounted at `/home` and `/var`. - Each VM gets one sparse writable overlay file (`cow.ext4`) that stores its changes on top of the shared base image.
- `run.sh` seeds those `/home` and `/var` disks from the rootfs snapshot before boot so the guest sees the base image contents there on first boot. - `/root` and `/var` live inside that per-VM overlay, so VMs can install packages without copying separate disks per VM.
- The base image must include `/etc/fstab` entries for `/dev/vdb``/home` and `/dev/vdc``/var`. - `run.sh` masks stale `home.mount` and `var.mount` units at boot so older images with `/dev/vdb` and `/dev/vdc` entries in `/etc/fstab` still boot.
- `/run` and `/tmp` should be tmpfs via `/etc/fstab`. - `/run` and `/tmp` should be tmpfs via `/etc/fstab`.
## SSH ## SSH
@ -84,6 +84,9 @@ preloaded so Docker works out of the box. Pass the base rootfs as a positional
argument; the output defaults to `docker-<base filename>` in the same directory argument; the output defaults to `docker-<base filename>` in the same directory
unless you pass `--out`. unless you pass `--out`.
Base guest packages come from `./packages.apt`. Edit that file to bake tools
like `vim` and `tmux` into rebuilt images.
``` ```
./customize.sh ./rootfs.ext4 --size 6G --docker ./customize.sh ./rootfs.ext4 --size 6G --docker
``` ```
@ -114,6 +117,16 @@ invoke `make-rootfs.sh` to build it.
`make-rootfs.sh` chooses the first available base image: `make-rootfs.sh` chooses the first available base image:
- `./rootfs.ext4` - `./rootfs.ext4`
If `./packages.apt` changes after `rootfs-docker.ext4` is built, `run.sh` will
warn and keep using the existing image. `make-rootfs.sh` will also warn and
exit without rebuilding while the image already exists.
To rebuild after package changes:
```
rm -f ./rootfs-docker.ext4 ./rootfs-docker.ext4.packages.sha256
./make-rootfs.sh
```
## Interactive Customization ## Interactive Customization
To create a writable copy and customize it manually over SSH (no automatic To create a writable copy and customize it manually over SSH (no automatic
package/config changes), use: package/config changes), use:

View file

@ -31,6 +31,7 @@ parse_size() {
DIR="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)" DIR="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)"
source "$DIR/dns.sh" source "$DIR/dns.sh"
source "$DIR/packages.sh"
STATE="$DIR/state" STATE="$DIR/state"
VM_ROOT="$STATE/vms" VM_ROOT="$STATE/vms"
mkdir -p "$VM_ROOT" mkdir -p "$VM_ROOT"
@ -52,6 +53,7 @@ OUT_ROOTFS=""
SIZE_SPEC="" SIZE_SPEC=""
INSTALL_DOCKER=0 INSTALL_DOCKER=0
MODULES_DIR="$DIR/wtf/root/lib/modules/6.8.0-94-generic" MODULES_DIR="$DIR/wtf/root/lib/modules/6.8.0-94-generic"
PACKAGES_FILE="$(banger_packages_file)"
while [[ $# -gt 0 ]]; do while [[ $# -gt 0 ]]; do
case "$1" in case "$1" in
--out) --out)
@ -136,6 +138,25 @@ if ! command -v jq >/dev/null 2>&1; then
log "jq required" log "jq required"
exit 1 exit 1
fi fi
if ! command -v sha256sum >/dev/null 2>&1; then
log "sha256sum required to record package manifest metadata"
exit 1
fi
if [[ ! -f "$PACKAGES_FILE" ]]; then
log "package manifest not found: $PACKAGES_FILE"
exit 1
fi
APT_PACKAGES=()
if ! banger_packages_read_array APT_PACKAGES "$PACKAGES_FILE"; then
log "package manifest is empty: $PACKAGES_FILE"
exit 1
fi
if ! PACKAGES_HASH="$(printf '%s\n' "${APT_PACKAGES[@]}" | banger_packages_hash_stream)"; then
log "failed to hash package manifest: $PACKAGES_FILE"
exit 1
fi
printf -v APT_PACKAGES_ESCAPED '%q ' "${APT_PACKAGES[@]}"
log "copying base rootfs to $OUT_ROOTFS" log "copying base rootfs to $OUT_ROOTFS"
cp --reflink=auto "$BASE_ROOTFS" "$OUT_ROOTFS" cp --reflink=auto "$BASE_ROOTFS" "$OUT_ROOTFS"
@ -223,7 +244,7 @@ sudo -E curl --unix-socket "$API_SOCK" -X PUT http://localhost/machine-config \
"smt": false "smt": false
}' >/dev/null }' >/dev/null
KCMD="console=ttyS0 reboot=k panic=1 pci=off root=/dev/vda rw ip=${GUEST_IP}::${BR_IP}:255.255.255.0:${VM_NAME}:eth0:off:${DNS_SERVER} hostname=${VM_NAME}" KCMD="console=ttyS0 reboot=k panic=1 pci=off root=/dev/vda rw ip=${GUEST_IP}::${BR_IP}:255.255.255.0:${VM_NAME}:eth0:off:${DNS_SERVER} hostname=${VM_NAME} systemd.mask=home.mount systemd.mask=var.mount"
INITRD_JSON="" INITRD_JSON=""
if [[ -n "$INITRD" ]]; then if [[ -n "$INITRD" ]]; then
@ -286,13 +307,19 @@ log "enabling NAT for customization"
sudo -E ./nat.sh up "$VM_TAG" >/dev/null sudo -E ./nat.sh up "$VM_TAG" >/dev/null
log "waiting for SSH" log "waiting for SSH"
SSH_READY=0
for _ in $(seq 1 60); do for _ in $(seq 1 60); do
if ssh -i "$SSH_KEY" -o StrictHostKeyChecking=no -o UserKnownHostsFile=/dev/null \ if ssh -i "$SSH_KEY" -o StrictHostKeyChecking=no -o UserKnownHostsFile=/dev/null \
"root@${GUEST_IP}" "true" >/dev/null 2>&1; then "root@${GUEST_IP}" "true" >/dev/null 2>&1; then
SSH_READY=1
break break
fi fi
sleep 1 sleep 1
done done
if [[ "$SSH_READY" -ne 1 ]]; then
log "ssh did not become ready on $GUEST_IP"
exit 1
fi
log "configuring guest" log "configuring guest"
ssh -i "$SSH_KEY" -o StrictHostKeyChecking=no -o UserKnownHostsFile=/dev/null \ ssh -i "$SSH_KEY" -o StrictHostKeyChecking=no -o UserKnownHostsFile=/dev/null \
@ -300,13 +327,8 @@ ssh -i "$SSH_KEY" -o StrictHostKeyChecking=no -o UserKnownHostsFile=/dev/null \
printf 'nameserver %s\n' \"$DNS_SERVER\" > /etc/resolv.conf printf 'nameserver %s\n' \"$DNS_SERVER\" > /etc/resolv.conf
echo \"$VM_NAME\" > /etc/hostname echo \"$VM_NAME\" > /etc/hostname
printf '127.0.0.1 localhost\n127.0.1.1 %s\n' \"$VM_NAME\" > /etc/hosts printf '127.0.0.1 localhost\n127.0.1.1 %s\n' \"$VM_NAME\" > /etc/hosts
mkdir -p /home /var touch /etc/fstab
if ! grep -q '^/dev/vdb ' /etc/fstab; then sed -i '\|^/dev/vdb[[:space:]]\+/home[[:space:]]|d; \|^/dev/vdc[[:space:]]\+/var[[:space:]]|d' /etc/fstab
echo '/dev/vdb /home ext4 defaults 0 2' >> /etc/fstab
fi
if ! grep -q '^/dev/vdc ' /etc/fstab; then
echo '/dev/vdc /var ext4 defaults 0 2' >> /etc/fstab
fi
if ! grep -q '^tmpfs /run ' /etc/fstab; then if ! grep -q '^tmpfs /run ' /etc/fstab; then
echo 'tmpfs /run tmpfs defaults,nodev,nosuid,mode=0755 0 0' >> /etc/fstab echo 'tmpfs /run tmpfs defaults,nodev,nosuid,mode=0755 0 0' >> /etc/fstab
fi fi
@ -315,7 +337,7 @@ if ! grep -q '^tmpfs /tmp ' /etc/fstab; then
fi fi
apt-get update apt-get update
DEBIAN_FRONTEND=noninteractive apt-get -y upgrade DEBIAN_FRONTEND=noninteractive apt-get -y upgrade
DEBIAN_FRONTEND=noninteractive apt-get -y install git less tree ca-certificates curl DEBIAN_FRONTEND=noninteractive apt-get -y install ${APT_PACKAGES_ESCAPED}
if [[ \"$INSTALL_DOCKER\" == \"1\" ]]; then if [[ \"$INSTALL_DOCKER\" == \"1\" ]]; then
DEBIAN_FRONTEND=noninteractive apt-get -y remove containerd || true DEBIAN_FRONTEND=noninteractive apt-get -y remove containerd || true
if ! DEBIAN_FRONTEND=noninteractive apt-get -y install docker-ce docker-ce-cli containerd.io docker-buildx-plugin docker-compose-plugin; then if ! DEBIAN_FRONTEND=noninteractive apt-get -y install docker-ce docker-ce-cli containerd.io docker-buildx-plugin docker-compose-plugin; then
@ -362,4 +384,5 @@ for _ in $(seq 1 200); do
fi fi
sleep 0.05 sleep 0.05
done done
banger_write_rootfs_manifest_metadata "$OUT_ROOTFS" "$PACKAGES_HASH"
log "done" log "done"

View file

@ -7,7 +7,7 @@ log() {
usage() { usage() {
cat <<'EOF' cat <<'EOF'
Usage: ./interactive.sh <base-rootfs> [--out <path>] [--size <size>] [--home-size <size>] [--var-size <size>] Usage: ./interactive.sh <base-rootfs> [--out <path>] [--size <size>]
Creates a writable copy of the base rootfs and boots a VM so you can Creates a writable copy of the base rootfs and boots a VM so you can
customize it manually over SSH. No automatic package/config changes customize it manually over SSH. No automatic package/config changes
@ -45,14 +45,10 @@ BR_DEV="br-fc"
BR_IP="172.16.0.1" BR_IP="172.16.0.1"
CIDR="24" CIDR="24"
DNS_SERVER="1.1.1.1" DNS_SERVER="1.1.1.1"
DEFAULT_HOME_SIZE="2G"
DEFAULT_VAR_SIZE="2G"
BASE_ROOTFS="" BASE_ROOTFS=""
OUT_ROOTFS="" OUT_ROOTFS=""
SIZE_SPEC="" SIZE_SPEC=""
HOME_SIZE="$DEFAULT_HOME_SIZE"
VAR_SIZE="$DEFAULT_VAR_SIZE"
while [[ $# -gt 0 ]]; do while [[ $# -gt 0 ]]; do
case "$1" in case "$1" in
--out) --out)
@ -63,14 +59,6 @@ while [[ $# -gt 0 ]]; do
SIZE_SPEC="${2:-}" SIZE_SPEC="${2:-}"
shift 2 shift 2
;; ;;
--home-size)
HOME_SIZE="${2:-}"
shift 2
;;
--var-size)
VAR_SIZE="${2:-}"
shift 2
;;
-h|--help) -h|--help)
usage usage
exit 0 exit 0
@ -115,9 +103,6 @@ if [[ -e "$OUT_ROOTFS" ]]; then
exit 1 exit 1
fi fi
HOME_BYTES="$(parse_size "$HOME_SIZE")" || { log "invalid --home-size value: $HOME_SIZE"; exit 1; }
VAR_BYTES="$(parse_size "$VAR_SIZE")" || { log "invalid --var-size value: $VAR_SIZE"; exit 1; }
log "copying base rootfs to $OUT_ROOTFS" log "copying base rootfs to $OUT_ROOTFS"
cp --reflink=auto "$BASE_ROOTFS" "$OUT_ROOTFS" cp --reflink=auto "$BASE_ROOTFS" "$OUT_ROOTFS"
@ -143,8 +128,6 @@ mkdir -p "$VM_DIR"
API_SOCK="${XDG_RUNTIME_DIR:-/run/user/$(id -u)}/banger/fc-$VM_TAG.sock" API_SOCK="${XDG_RUNTIME_DIR:-/run/user/$(id -u)}/banger/fc-$VM_TAG.sock"
LOG_FILE="$VM_DIR/firecracker.log" LOG_FILE="$VM_DIR/firecracker.log"
TAP_DEV="tap-fc-$VM_TAG" TAP_DEV="tap-fc-$VM_TAG"
HOME_PATH="$VM_DIR/home.ext4"
VAR_PATH="$VM_DIR/var.ext4"
DNS_NAME="" DNS_NAME=""
# Allocate guest IP # Allocate guest IP
@ -185,15 +168,6 @@ sudo ip link set "$TAP_DEV" master "$BR_DEV"
sudo ip link set "$TAP_DEV" up sudo ip link set "$TAP_DEV" up
sudo ip link set "$BR_DEV" up sudo ip link set "$BR_DEV" up
if ! command -v mkfs.ext4 >/dev/null 2>&1; then
log "mkfs.ext4 required to create home/var disks"
exit 1
fi
truncate -s "$HOME_BYTES" "$HOME_PATH"
mkfs.ext4 -F "$HOME_PATH" >/dev/null
truncate -s "$VAR_BYTES" "$VAR_PATH"
mkfs.ext4 -F "$VAR_PATH" >/dev/null
log "starting firecracker process" log "starting firecracker process"
rm -f "$API_SOCK" rm -f "$API_SOCK"
nohup sudo -E "$FC_BIN" --api-sock "$API_SOCK" >"$LOG_FILE" 2>&1 & nohup sudo -E "$FC_BIN" --api-sock "$API_SOCK" >"$LOG_FILE" 2>&1 &
@ -215,7 +189,7 @@ sudo -E curl --unix-socket "$API_SOCK" -X PUT http://localhost/machine-config \
"smt": false "smt": false
}' >/dev/null }' >/dev/null
KCMD="console=ttyS0 reboot=k panic=1 pci=off root=/dev/vda rw ip=${GUEST_IP}::${BR_IP}:255.255.255.0:${VM_NAME}:eth0:off:${DNS_SERVER} hostname=${VM_NAME}" KCMD="console=ttyS0 reboot=k panic=1 pci=off root=/dev/vda rw ip=${GUEST_IP}::${BR_IP}:255.255.255.0:${VM_NAME}:eth0:off:${DNS_SERVER} hostname=${VM_NAME} systemd.mask=home.mount systemd.mask=var.mount"
sudo -E curl --unix-socket "$API_SOCK" -X PUT http://localhost/boot-source \ sudo -E curl --unix-socket "$API_SOCK" -X PUT http://localhost/boot-source \
-H "Content-Type: application/json" \ -H "Content-Type: application/json" \
@ -234,24 +208,6 @@ sudo -E curl --unix-socket "$API_SOCK" -X PUT http://localhost/drives/rootfs \
\"is_read_only\": false \"is_read_only\": false
}" >/dev/null }" >/dev/null
sudo -E curl --unix-socket "$API_SOCK" -X PUT http://localhost/drives/home \
-H "Content-Type: application/json" \
-d "{
\"drive_id\": \"home\",
\"path_on_host\": \"$HOME_PATH\",
\"is_root_device\": false,
\"is_read_only\": false
}" >/dev/null
sudo -E curl --unix-socket "$API_SOCK" -X PUT http://localhost/drives/var \
-H "Content-Type: application/json" \
-d "{
\"drive_id\": \"var\",
\"path_on_host\": \"$VAR_PATH\",
\"is_root_device\": false,
\"is_read_only\": false
}" >/dev/null
sudo -E curl --unix-socket "$API_SOCK" -X PUT http://localhost/network-interfaces/eth0 \ sudo -E curl --unix-socket "$API_SOCK" -X PUT http://localhost/network-interfaces/eth0 \
-H "Content-Type: application/json" \ -H "Content-Type: application/json" \
-d "{ -d "{

5
make-rootfs.sh Normal file → Executable file
View file

@ -18,6 +18,7 @@ EOF
} }
DIR="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)" DIR="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)"
source "$DIR/packages.sh"
OUT_ROOTFS="$DIR/rootfs-docker.ext4" OUT_ROOTFS="$DIR/rootfs-docker.ext4"
SIZE_SPEC="6G" SIZE_SPEC="6G"
BASE_ROOTFS="" BASE_ROOTFS=""
@ -45,6 +46,10 @@ while [[ $# -gt 0 ]]; do
done done
if [[ -f "$OUT_ROOTFS" ]]; then if [[ -f "$OUT_ROOTFS" ]]; then
OUT_ROOTFS_WARNING="$(banger_rootfs_manifest_warning "$OUT_ROOTFS" || true)"
if [[ -n "$OUT_ROOTFS_WARNING" ]]; then
log "warning: $OUT_ROOTFS_WARNING"
fi
log "already exists: $OUT_ROOTFS" log "already exists: $OUT_ROOTFS"
exit 0 exit 0
fi fi

7
packages.apt Normal file
View file

@ -0,0 +1,7 @@
git
less
tree
ca-certificates
curl
vim
tmux

115
packages.sh Normal file
View file

@ -0,0 +1,115 @@
#!/usr/bin/env bash
BANGER_PACKAGES_DIR="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)"
BANGER_APT_PACKAGES_FILE="${BANGER_APT_PACKAGES_FILE:-$BANGER_PACKAGES_DIR/packages.apt}"
banger_packages_file() {
printf '%s' "$BANGER_APT_PACKAGES_FILE"
}
banger_packages_normalized_lines() {
local packages_file="${1:-$BANGER_APT_PACKAGES_FILE}"
[[ -f "$packages_file" ]] || return 1
awk '
{
sub(/\r$/, "")
sub(/[[:space:]]*#.*$/, "")
gsub(/^[[:space:]]+|[[:space:]]+$/, "")
if ($0 != "") print
}
' "$packages_file"
}
banger_packages_read_array() {
local -n out="$1"
local packages_file="${2:-$BANGER_APT_PACKAGES_FILE}"
mapfile -t out < <(banger_packages_normalized_lines "$packages_file")
(( ${#out[@]} > 0 ))
}
banger_packages_hash_stream() {
command -v sha256sum >/dev/null 2>&1 || return 1
sha256sum | awk '{print $1}'
}
banger_packages_manifest_hash() {
local packages_file="${1:-$BANGER_APT_PACKAGES_FILE}"
[[ -f "$packages_file" ]] || return 1
banger_packages_normalized_lines "$packages_file" | banger_packages_hash_stream
}
banger_rootfs_manifest_metadata_path() {
local rootfs_path="$1"
printf '%s.packages.sha256' "$rootfs_path"
}
banger_rootfs_manifest_recorded_hash() {
local rootfs_path="$1"
local metadata_file recorded_hash
metadata_file="$(banger_rootfs_manifest_metadata_path "$rootfs_path")"
[[ -f "$metadata_file" ]] || return 1
recorded_hash="$(head -n 1 "$metadata_file" | tr -d '[:space:]')"
[[ -n "$recorded_hash" ]] || return 1
printf '%s' "$recorded_hash"
}
banger_write_rootfs_manifest_metadata() {
local rootfs_path="$1"
local manifest_hash="$2"
local metadata_file
metadata_file="$(banger_rootfs_manifest_metadata_path "$rootfs_path")"
printf '%s\n' "$manifest_hash" > "$metadata_file"
}
banger_rootfs_manifest_status() {
local rootfs_path="$1"
local current_hash recorded_hash
if [[ ! -f "$rootfs_path" ]]; then
printf '%s' "missing-rootfs"
return 0
fi
if ! current_hash="$(banger_packages_manifest_hash)"; then
printf '%s' "unknown"
return 0
fi
if ! recorded_hash="$(banger_rootfs_manifest_recorded_hash "$rootfs_path")"; then
printf '%s' "missing-metadata"
return 0
fi
if [[ "$recorded_hash" == "$current_hash" ]]; then
printf '%s' "fresh"
else
printf '%s' "stale"
fi
}
banger_rootfs_manifest_warning() {
local rootfs_path="$1"
local status
status="$(banger_rootfs_manifest_status "$rootfs_path")"
case "$status" in
stale)
printf '%s was built with an older package manifest; rebuild it explicitly to pick up package changes' "$rootfs_path"
;;
missing-metadata)
printf '%s has no package manifest metadata; rebuild it explicitly to pick up package changes' "$rootfs_path"
;;
unknown)
printf 'unable to compare %s against %s; install sha256sum and verify the package manifest manually' "$rootfs_path" "$BANGER_APT_PACKAGES_FILE"
;;
*)
return 1
;;
esac
}

View file

@ -76,8 +76,6 @@ VM_NAME="$(jq -r '.meta.name // empty' "$VM_JSON")"
PID="$(jq -r '.meta.pid // empty' "$VM_JSON")" PID="$(jq -r '.meta.pid // empty' "$VM_JSON")"
ROOTFS="$(jq -r '.meta.rootfs // empty' "$VM_JSON")" ROOTFS="$(jq -r '.meta.rootfs // empty' "$VM_JSON")"
KERNEL="$(jq -r '.meta.kernel // empty' "$VM_JSON")" KERNEL="$(jq -r '.meta.kernel // empty' "$VM_JSON")"
HOME_PATH="$(jq -r '.meta.home_path // empty' "$VM_JSON")"
VAR_PATH="$(jq -r '.meta.var_path // empty' "$VM_JSON")"
TAP_DEV="$(jq -r '.meta.tap // empty' "$VM_JSON")" TAP_DEV="$(jq -r '.meta.tap // empty' "$VM_JSON")"
API_SOCK="$(jq -r '.meta.api_sock // empty' "$VM_JSON")" API_SOCK="$(jq -r '.meta.api_sock // empty' "$VM_JSON")"
LOG_FILE="$(jq -r '.meta.log // empty' "$VM_JSON")" LOG_FILE="$(jq -r '.meta.log // empty' "$VM_JSON")"
@ -90,11 +88,11 @@ COW_LOOP_OLD="$(jq -r '.meta.cow_loop // empty' "$VM_JSON")"
INITRD_PATH="$(jq -r '.config["boot-source"].initrd_path // empty' "$VM_JSON")" INITRD_PATH="$(jq -r '.config["boot-source"].initrd_path // empty' "$VM_JSON")"
DNS_NAME="$(banger_dns_name "$VM_NAME")" DNS_NAME="$(banger_dns_name "$VM_NAME")"
if [[ -z "$ROOTFS" || -z "$KERNEL" || -z "$HOME_PATH" || -z "$VAR_PATH" || -z "$API_SOCK" || -z "$TAP_DEV" || -z "$GUEST_IP" || -z "$DM_NAME" || -z "$COW_FILE" ]]; then if [[ -z "$ROOTFS" || -z "$KERNEL" || -z "$API_SOCK" || -z "$TAP_DEV" || -z "$GUEST_IP" || -z "$DM_NAME" || -z "$COW_FILE" ]]; then
log "vm.json missing required fields" log "vm.json missing required fields"
exit 1 exit 1
fi fi
if [[ ! -f "$ROOTFS" || ! -f "$KERNEL" || ! -f "$HOME_PATH" || ! -f "$VAR_PATH" || ! -f "$COW_FILE" || ! -f "$FC_BIN" ]]; then if [[ ! -f "$ROOTFS" || ! -f "$KERNEL" || ! -f "$COW_FILE" || ! -f "$FC_BIN" ]]; then
log "missing disk/kernel file(s)" log "missing disk/kernel file(s)"
exit 1 exit 1
fi fi
@ -203,9 +201,9 @@ log "configuring machine"
-d "$(jq -c '.config["machine-config"]' "$VM_JSON")" >/dev/null -d "$(jq -c '.config["machine-config"]' "$VM_JSON")" >/dev/null
boot_args="$(jq -r '.config["boot-source"].boot_args // empty' "$VM_JSON")" boot_args="$(jq -r '.config["boot-source"].boot_args // empty' "$VM_JSON")"
boot_args="$(printf '%s' "$boot_args" | sed -E 's/(^| )hostname=[^ ]+//g; s/(^| )ip=[^ ]+//g' | awk '{$1=$1; print}')" boot_args="$(printf '%s' "$boot_args" | sed -E 's/(^| )hostname=[^ ]+//g; s/(^| )ip=[^ ]+//g; s/(^| )systemd\.mask=home\.mount//g; s/(^| )systemd\.mask=var\.mount//g' | awk '{$1=$1; print}')"
boot_args="$boot_args ip=${GUEST_IP}::${BR_IP}:255.255.255.0::eth0:off:${DNS_SERVER}" boot_args="$boot_args ip=${GUEST_IP}::${BR_IP}:255.255.255.0::eth0:off:${DNS_SERVER}"
boot_args="$boot_args hostname=$VM_NAME" boot_args="$boot_args hostname=$VM_NAME systemd.mask=home.mount systemd.mask=var.mount"
INITRD_JSON="" INITRD_JSON=""
if [[ -n "$INITRD_PATH" ]]; then if [[ -n "$INITRD_PATH" ]]; then
INITRD_JSON=", \"initrd_path\": \"$INITRD_PATH\"" INITRD_JSON=", \"initrd_path\": \"$INITRD_PATH\""
@ -229,24 +227,6 @@ log "attaching drives"
\"is_read_only\": false \"is_read_only\": false
}" >/dev/null }" >/dev/null
/usr/bin/sudo /usr/bin/curl --unix-socket "$API_SOCK" -X PUT http://localhost/drives/home \
-H "Content-Type: application/json" \
-d "{
\"drive_id\": \"home\",
\"path_on_host\": \"$HOME_PATH\",
\"is_root_device\": false,
\"is_read_only\": false
}" >/dev/null
/usr/bin/sudo /usr/bin/curl --unix-socket "$API_SOCK" -X PUT http://localhost/drives/var \
-H "Content-Type: application/json" \
-d "{
\"drive_id\": \"var\",
\"path_on_host\": \"$VAR_PATH\",
\"is_root_device\": false,
\"is_read_only\": false
}" >/dev/null
log "configuring network interface" log "configuring network interface"
/usr/bin/sudo /usr/bin/curl --unix-socket "$API_SOCK" -X PUT http://localhost/network-interfaces/eth0 \ /usr/bin/sudo /usr/bin/curl --unix-socket "$API_SOCK" -X PUT http://localhost/network-interfaces/eth0 \
-H "Content-Type: application/json" \ -H "Content-Type: application/json" \

141
run.sh
View file

@ -13,11 +13,10 @@ Options:
--name <name> VM name (lowercase letters, digits, -) --name <name> VM name (lowercase letters, digits, -)
--vcpu <count> vCPU count (default: 2) --vcpu <count> vCPU count (default: 2)
--ram <mib> RAM in MiB (default: 1024) --ram <mib> RAM in MiB (default: 1024)
--rootfs <path> Root filesystem image (default: ./rootfs.ext4) --overlay-size <size> Writable overlay size (e.g. 8G, 16384M)
--rootfs <path> Root filesystem image (default: ./rootfs-docker.ext4)
--kernel <path> Kernel image (default: ./vmlinux) --kernel <path> Kernel image (default: ./vmlinux)
--initrd <path> Initrd image (optional) --initrd <path> Initrd image (optional)
--home-size <size> Home disk size (e.g. 4G, 10240M)
--var-size <size> Var disk size (e.g. 4G, 10240M)
-h, --help Show this help -h, --help Show this help
EOF EOF
} }
@ -26,6 +25,7 @@ log "starting"
DIR="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)" DIR="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)"
source "$DIR/dns.sh" source "$DIR/dns.sh"
source "$DIR/packages.sh"
STATE="$DIR/state" STATE="$DIR/state"
VM_ROOT="$STATE/vms" VM_ROOT="$STATE/vms"
mkdir -p "$VM_ROOT" mkdir -p "$VM_ROOT"
@ -43,20 +43,17 @@ CIDR="24"
DEFAULT_VCPU=2 DEFAULT_VCPU=2
DEFAULT_RAM=1024 DEFAULT_RAM=1024
DEFAULT_HOME_SIZE="2G" DEFAULT_OVERLAY_SIZE="8G"
DEFAULT_VAR_SIZE="2G"
MIN_VCPU=1 MIN_VCPU=1
MAX_VCPU=16 MAX_VCPU=16
MIN_RAM=256 MIN_RAM=256
MAX_RAM=32768 MAX_RAM=32768
MAX_DISK_BYTES=$((128 * 1024 * 1024 * 1024)) MAX_DISK_BYTES=$((128 * 1024 * 1024 * 1024))
DNS_SERVER="1.1.1.1" DNS_SERVER="1.1.1.1"
COW_SIZE="2G"
VCPU_COUNT="$DEFAULT_VCPU" VCPU_COUNT="$DEFAULT_VCPU"
RAM_MIB="$DEFAULT_RAM" RAM_MIB="$DEFAULT_RAM"
HOME_SIZE="$DEFAULT_HOME_SIZE" OVERLAY_SIZE="$DEFAULT_OVERLAY_SIZE"
VAR_SIZE="$DEFAULT_VAR_SIZE"
KERNEL="$DEFAULT_KERNEL" KERNEL="$DEFAULT_KERNEL"
ROOTFS="$DEFAULT_ROOTFS" ROOTFS="$DEFAULT_ROOTFS"
INITRD="$DEFAULT_INITRD" INITRD="$DEFAULT_INITRD"
@ -105,6 +102,10 @@ while [[ $# -gt 0 ]]; do
RAM_MIB="${2:-}" RAM_MIB="${2:-}"
shift 2 shift 2
;; ;;
--overlay-size)
OVERLAY_SIZE="${2:-}"
shift 2
;;
--rootfs) --rootfs)
ROOTFS="${2:-}" ROOTFS="${2:-}"
shift 2 shift 2
@ -117,14 +118,6 @@ while [[ $# -gt 0 ]]; do
INITRD="${2:-}" INITRD="${2:-}"
shift 2 shift 2
;; ;;
--home-size)
HOME_SIZE="${2:-}"
shift 2
;;
--var-size)
VAR_SIZE="${2:-}"
shift 2
;;
-h|--help) -h|--help)
usage usage
exit 0 exit 0
@ -155,23 +148,12 @@ if (( RAM_MIB < MIN_RAM || RAM_MIB > MAX_RAM )); then
exit 1 exit 1
fi fi
HOME_BYTES="" if ! OVERLAY_BYTES="$(parse_disk_size "$OVERLAY_SIZE")"; then
if ! HOME_BYTES="$(parse_disk_size "$HOME_SIZE")"; then log "invalid --overlay-size value: $OVERLAY_SIZE"
log "invalid --home-size value: $HOME_SIZE"
exit 1 exit 1
fi fi
if (( HOME_BYTES > MAX_DISK_BYTES )); then if (( OVERLAY_BYTES > MAX_DISK_BYTES )); then
log "home-size exceeds max of $((MAX_DISK_BYTES / 1024 / 1024 / 1024))G" log "overlay-size exceeds max of $((MAX_DISK_BYTES / 1024 / 1024 / 1024))G"
exit 1
fi
VAR_BYTES=""
if ! VAR_BYTES="$(parse_disk_size "$VAR_SIZE")"; then
log "invalid --var-size value: $VAR_SIZE"
exit 1
fi
if (( VAR_BYTES > MAX_DISK_BYTES )); then
log "var-size exceeds max of $((MAX_DISK_BYTES / 1024 / 1024 / 1024))G"
exit 1 exit 1
fi fi
@ -207,6 +189,12 @@ if [[ ! -f "$ROOTFS" ]]; then
log "rootfs not found: $ROOTFS" log "rootfs not found: $ROOTFS"
exit 1 exit 1
fi fi
if [[ "$ROOTFS" == "$DEFAULT_ROOTFS" ]]; then
ROOTFS_WARNING="$(banger_rootfs_manifest_warning "$ROOTFS" || true)"
if [[ -n "$ROOTFS_WARNING" ]]; then
log "warning: $ROOTFS_WARNING"
fi
fi
if [[ ! -f "$KERNEL" ]]; then if [[ ! -f "$KERNEL" ]]; then
log "kernel not found: $KERNEL" log "kernel not found: $KERNEL"
exit 1 exit 1
@ -248,49 +236,13 @@ sudo -v
VM_STARTED=0 VM_STARTED=0
CLEANUP_ON_EXIT=0 CLEANUP_ON_EXIT=0
KEEP_VM_DIR_ON_FAIL=1 KEEP_VM_DIR_ON_FAIL=1
HOME_PATH="$VM_DIR/home.ext4"
VAR_PATH="$VM_DIR/var.ext4"
COW_FILE="$VM_DIR/cow.ext4" COW_FILE="$VM_DIR/cow.ext4"
BASE_LOOP="" BASE_LOOP=""
COW_LOOP="" COW_LOOP=""
DM_NAME="fc-rootfs-$VM_TAG" DM_NAME="fc-rootfs-$VM_TAG"
DM_DEV="" DM_DEV=""
SEED_ROOT_MNT="$VM_DIR/mnt-root"
SEED_HOME_MNT="$VM_DIR/mnt-home"
SEED_VAR_MNT="$VM_DIR/mnt-var"
DNS_NAME="" DNS_NAME=""
unmount_if_needed() {
local mount_path="$1"
[[ -n "$mount_path" ]] || return 0
sudo umount "$mount_path" 2>/dev/null || true
}
seed_volume_from_rootfs() {
local source_dir="$1"
local target_image="$2"
local target_mount="$3"
local label="$4"
if [[ ! -d "$source_dir" ]]; then
log "source directory missing in rootfs snapshot: $label"
exit 1
fi
mkdir -p "$target_mount"
sudo mount "$target_image" "$target_mount"
sudo cp -a "$source_dir/." "$target_mount/"
sudo umount "$target_mount"
}
populate_data_disks() {
mkdir -p "$SEED_ROOT_MNT" "$SEED_HOME_MNT" "$SEED_VAR_MNT"
sudo mount -o ro "$DM_DEV" "$SEED_ROOT_MNT"
seed_volume_from_rootfs "$SEED_ROOT_MNT/home" "$HOME_PATH" "$SEED_HOME_MNT" "/home"
seed_volume_from_rootfs "$SEED_ROOT_MNT/var" "$VAR_PATH" "$SEED_VAR_MNT" "/var"
sudo umount "$SEED_ROOT_MNT"
}
cleanup() { cleanup() {
local exit_code=$? local exit_code=$?
if [[ "$VM_STARTED" -eq 1 && "$CLEANUP_ON_EXIT" -eq 0 ]]; then if [[ "$VM_STARTED" -eq 1 && "$CLEANUP_ON_EXIT" -eq 0 ]]; then
@ -304,9 +256,6 @@ cleanup() {
sudo ip link del "$TAP_DEV" 2>/dev/null || true sudo ip link del "$TAP_DEV" 2>/dev/null || true
fi fi
rm -f "${API_SOCK:-}" rm -f "${API_SOCK:-}"
unmount_if_needed "${SEED_VAR_MNT:-}"
unmount_if_needed "${SEED_HOME_MNT:-}"
unmount_if_needed "${SEED_ROOT_MNT:-}"
banger_dns_remove_record_name "${DNS_NAME:-}" banger_dns_remove_record_name "${DNS_NAME:-}"
if [[ -n "${DM_NAME:-}" ]]; then if [[ -n "${DM_NAME:-}" ]]; then
sudo dmsetup remove "$DM_NAME" 2>/dev/null || true sudo dmsetup remove "$DM_NAME" 2>/dev/null || true
@ -351,15 +300,6 @@ else
log "setcap not available; firecracker may need root to open TAP" log "setcap not available; firecracker may need root to open TAP"
fi fi
if ! command -v mkfs.ext4 >/dev/null 2>&1; then
log "mkfs.ext4 required to create home/var disks"
exit 1
fi
truncate -s "$HOME_BYTES" "$HOME_PATH"
mkfs.ext4 -F "$HOME_PATH" >/dev/null
truncate -s "$VAR_BYTES" "$VAR_PATH"
mkfs.ext4 -F "$VAR_PATH" >/dev/null
if ! command -v dmsetup >/dev/null 2>&1 || ! command -v losetup >/dev/null 2>&1 || ! command -v blockdev >/dev/null 2>&1; then if ! command -v dmsetup >/dev/null 2>&1 || ! command -v losetup >/dev/null 2>&1 || ! command -v blockdev >/dev/null 2>&1; then
log "dmsetup, losetup, and blockdev are required for rootfs snapshots" log "dmsetup, losetup, and blockdev are required for rootfs snapshots"
exit 1 exit 1
@ -368,23 +308,13 @@ if ! command -v e2cp >/dev/null 2>&1 || ! command -v e2rm >/dev/null 2>&1; then
log "e2cp and e2rm are required to set hostname and resolv.conf" log "e2cp and e2rm are required to set hostname and resolv.conf"
exit 1 exit 1
fi fi
if ! command -v mount >/dev/null 2>&1 || ! command -v umount >/dev/null 2>&1; then
log "mount and umount are required to populate home/var disks"
exit 1
fi
if ! command -v jq >/dev/null 2>&1; then if ! command -v jq >/dev/null 2>&1; then
log "jq is required to persist VM metadata" log "jq is required to persist VM metadata"
exit 1 exit 1
fi fi
COW_BYTES="$(parse_disk_size "$COW_SIZE")"
if [[ -z "$COW_BYTES" ]]; then
log "invalid COW size: $COW_SIZE"
exit 1
fi
BASE_LOOP="$(sudo losetup -f --show --read-only "$ROOTFS")" BASE_LOOP="$(sudo losetup -f --show --read-only "$ROOTFS")"
truncate -s "$COW_BYTES" "$COW_FILE" truncate -s "$OVERLAY_BYTES" "$COW_FILE"
COW_LOOP="$(sudo losetup -f --show "$COW_FILE")" COW_LOOP="$(sudo losetup -f --show "$COW_FILE")"
SECTORS="$(sudo blockdev --getsz "$BASE_LOOP")" SECTORS="$(sudo blockdev --getsz "$BASE_LOOP")"
sudo dmsetup create "$DM_NAME" --table "0 $SECTORS snapshot $BASE_LOOP $COW_LOOP P 8" sudo dmsetup create "$DM_NAME" --table "0 $SECTORS snapshot $BASE_LOOP $COW_LOOP P 8"
@ -412,9 +342,6 @@ sudo e2cp "$HOSTS_TMP" "$DM_DEV:/etc/hosts" >/dev/null 2>&1 || {
exit 1 exit 1
} }
log "populating /home and /var disks from rootfs snapshot"
populate_data_disks
# Host bridge # Host bridge
if ! ip link show "$BR_DEV" >/dev/null 2>&1; then if ! ip link show "$BR_DEV" >/dev/null 2>&1; then
log "creating host bridge $BR_DEV ($BR_IP/$CIDR)" log "creating host bridge $BR_DEV ($BR_IP/$CIDR)"
@ -480,7 +407,7 @@ log "configuring machine"
# Boot source # Boot source
log "configuring boot source" log "configuring boot source"
KCMD="console=ttyS0 reboot=k panic=1 pci=off root=/dev/vda rw ip=${GUEST_IP}::${BR_IP}:255.255.255.0::eth0:off:${DNS_SERVER} hostname=${VM_NAME}" KCMD="console=ttyS0 reboot=k panic=1 pci=off root=/dev/vda rw ip=${GUEST_IP}::${BR_IP}:255.255.255.0::eth0:off:${DNS_SERVER} hostname=${VM_NAME} systemd.mask=home.mount systemd.mask=var.mount"
INITRD_JSON="" INITRD_JSON=""
if [[ -n "$INITRD" ]]; then if [[ -n "$INITRD" ]]; then
@ -505,28 +432,6 @@ log "attaching root filesystem"
\"is_read_only\": false \"is_read_only\": false
}" >/dev/null }" >/dev/null
# Home filesystem
log "attaching home filesystem"
"${CURL_CMD[@]}" --unix-socket "$API_SOCK" -X PUT http://localhost/drives/home \
-H "Content-Type: application/json" \
-d "{
\"drive_id\": \"home\",
\"path_on_host\": \"$HOME_PATH\",
\"is_root_device\": false,
\"is_read_only\": false
}" >/dev/null
# Var filesystem
log "attaching var filesystem"
"${CURL_CMD[@]}" --unix-socket "$API_SOCK" -X PUT http://localhost/drives/var \
-H "Content-Type: application/json" \
-d "{
\"drive_id\": \"var\",
\"path_on_host\": \"$VAR_PATH\",
\"is_root_device\": false,
\"is_read_only\": false
}" >/dev/null
# Network interface # Network interface
log "configuring network interface" log "configuring network interface"
"${CURL_CMD[@]}" --unix-socket "$API_SOCK" -X PUT http://localhost/network-interfaces/eth0 \ "${CURL_CMD[@]}" --unix-socket "$API_SOCK" -X PUT http://localhost/network-interfaces/eth0 \
@ -556,8 +461,6 @@ jq -n \
--arg log "$LOG_FILE" \ --arg log "$LOG_FILE" \
--arg rootfs "$ROOTFS" \ --arg rootfs "$ROOTFS" \
--arg kernel "$KERNEL" \ --arg kernel "$KERNEL" \
--arg home_path "$HOME_PATH" \
--arg var_path "$VAR_PATH" \
--arg base_loop "$BASE_LOOP" \ --arg base_loop "$BASE_LOOP" \
--arg cow_file "$COW_FILE" \ --arg cow_file "$COW_FILE" \
--arg cow_loop "$COW_LOOP" \ --arg cow_loop "$COW_LOOP" \
@ -565,7 +468,7 @@ jq -n \
--arg dm_dev "$DM_DEV" \ --arg dm_dev "$DM_DEV" \
--arg dns_name "$DNS_NAME" \ --arg dns_name "$DNS_NAME" \
--argjson config "$VM_CONFIG_JSON" \ --argjson config "$VM_CONFIG_JSON" \
'{meta:{id:$id,name:$name,pid:$pid,created_at:$created_at,guest_ip:$guest_ip,tap:$tap,api_sock:$api_sock,log:$log,rootfs:$rootfs,kernel:$kernel,home_path:$home_path,var_path:$var_path,base_loop:$base_loop,cow_file:$cow_file,cow_loop:$cow_loop,dm_name:$dm_name,dm_dev:$dm_dev,dns_name:$dns_name},config:$config}' \ '{meta:{id:$id,name:$name,pid:$pid,created_at:$created_at,guest_ip:$guest_ip,tap:$tap,api_sock:$api_sock,log:$log,rootfs:$rootfs,kernel:$kernel,base_loop:$base_loop,cow_file:$cow_file,cow_loop:$cow_loop,dm_name:$dm_name,dm_dev:$dm_dev,dns_name:$dns_name},config:$config}' \
> "$VM_DIR/vm.json" > "$VM_DIR/vm.json"
VM_STARTED=1 VM_STARTED=1