Thales Maciel
f779b71e1b
Stop letting the explicit compile step overstate its coverage. The old py_compile globs only touched top-level modules, so syntax errors in nested packages could slip past make check and release-check.\n\nAdd a shared compile-check recipe in Makefile that runs python -m compileall -q src tests, and have both check and release-check use it so the local verification paths stay aligned. Update the GitHub Actions compile step and the matching runtime validation evidence doc to describe the same recursive compile contract.\n\nValidate with python3 -m compileall -q src tests, make check, and make release-check.
50 lines
3.9 KiB
Markdown
50 lines
3.9 KiB
Markdown
# Runtime Validation Report
|
|
|
|
This document tracks milestone 3 evidence for runtime reliability and
|
|
diagnostics.
|
|
|
|
## Automated evidence
|
|
|
|
Completed on 2026-03-12:
|
|
|
|
- `PYTHONPATH=src python3 -m unittest tests.test_diagnostics tests.test_aman_cli tests.test_aman tests.test_aiprocess`
|
|
- covers `doctor` versus `self-check`, tri-state diagnostic output, warning
|
|
versus failure exit codes, read-only model cache probing, and actionable
|
|
runtime log wording for audio, hotkey, injection, editor, and startup
|
|
failures
|
|
- `PYTHONPATH=src python3 -m unittest discover -s tests -p 'test_*.py'`
|
|
- confirms the runtime and diagnostics changes do not regress the broader
|
|
daemon, CLI, config, and portable bundle flows
|
|
- `python3 -m compileall -q src tests`
|
|
- verifies the updated runtime, diagnostics, and nested package modules
|
|
compile cleanly
|
|
|
|
## Automated scenario coverage
|
|
|
|
| Scenario | Evidence | Status | Notes |
|
|
| --- | --- | --- | --- |
|
|
| `doctor` and `self-check` have distinct roles | `tests.test_diagnostics`, `tests.test_aman_cli` | Complete | `self-check` extends `doctor` with service/model/startup readiness checks |
|
|
| Missing config remains read-only | `tests.test_diagnostics` | Complete | Missing config yields `warn` and does not write a default file |
|
|
| Managed model cache probing is read-only | `tests.test_diagnostics`, `tests.test_aiprocess` | Complete | `self-check` uses cache probing and does not download or repair |
|
|
| Warning-only diagnostics exit `0`; failures exit `2` | `tests.test_aman_cli` | Complete | Human and JSON output share the same status model |
|
|
| Runtime failures log stable IDs and one next step | `tests.test_aman_cli`, `tests.test_aman` | Complete | Covers hotkey, audio-input, injection, editor, and startup failure wording |
|
|
| Repeated start/stop and shutdown return to `idle` | `tests.test_aman` | Complete | Current daemon tests cover start, stop, cancel, pause, and shutdown paths |
|
|
|
|
## Manual X11 validation
|
|
|
|
One representative distro family with real runtime validation is enough to
|
|
close milestone 3 for now. Full Debian/Ubuntu, Arch, Fedora, and openSUSE
|
|
coverage remains a milestone 5 GA signoff requirement.
|
|
|
|
Store raw evidence for each runtime validation pass in
|
|
`user-readiness/<linux-timestamp>.md` and reference that file in the `Notes`
|
|
column.
|
|
|
|
| Scenario | Debian/Ubuntu | Arch | Fedora | openSUSE | Reviewer | Status | Notes |
|
|
| --- | --- | --- | --- | --- | --- | --- | --- |
|
|
| Service restart after a successful install | Pending | Pass | Pending | Pending | User | Arch validated | User-reported Arch X11 validation in [`1773357669.md`](../../user-readiness/1773357669.md); verify `systemctl --user restart aman` returns to the tray/ready state |
|
|
| Reboot followed by successful reuse | Pending | Pass | Pending | Pending | User | Arch validated | User-reported Arch X11 validation in [`1773357669.md`](../../user-readiness/1773357669.md); validate recovery after a real session restart |
|
|
| Offline startup with an already-cached model | Pending | Pass | Pending | Pending | User | Arch validated | User-reported Arch X11 validation in [`1773357669.md`](../../user-readiness/1773357669.md); cached-model offline start succeeded |
|
|
| Missing runtime dependency recovery | Pending | Pass | Pending | Pending | User | Arch validated | User-reported Arch X11 validation in [`1773357669.md`](../../user-readiness/1773357669.md); diagnostics pointed to the fix |
|
|
| Tray-triggered diagnostics logging | Pending | Pass | Pending | Pending | User | Arch validated | User-reported Arch X11 validation in [`1773357669.md`](../../user-readiness/1773357669.md); `Run Diagnostics` matched the documented log path |
|
|
| Service-failure escalation path | Pending | Pass | Pending | Pending | User | Arch validated | User-reported Arch X11 validation in [`1773357669.md`](../../user-readiness/1773357669.md); `doctor` -> `self-check` -> `journalctl` -> `aman run --verbose` was sufficient |
|