Add Vosk keystroke eval tooling and findings
This commit is contained in:
parent
8c1f7c1e13
commit
510d280b74
15 changed files with 2219 additions and 0 deletions
5
exploration/vosk/keystrokes/.gitignore
vendored
Normal file
5
exploration/vosk/keystrokes/.gitignore
vendored
Normal file
|
|
@ -0,0 +1,5 @@
|
|||
literal/manifest.jsonl
|
||||
literal/samples/
|
||||
nato/manifest.jsonl
|
||||
nato/samples/
|
||||
eval_runs/
|
||||
31
exploration/vosk/keystrokes/findings.md
Normal file
31
exploration/vosk/keystrokes/findings.md
Normal file
|
|
@ -0,0 +1,31 @@
|
|||
# Vosk Keystroke Grammar Findings
|
||||
|
||||
- Date (UTC): 2026-02-28
|
||||
- Run ID: `run-20260228T200047Z`
|
||||
- Dataset size:
|
||||
- Literal grammar: 90 samples
|
||||
- NATO grammar: 90 samples
|
||||
- Intents: 9 (`ctrl|shift|ctrl+shift` x `d|b|p`)
|
||||
|
||||
## Results
|
||||
|
||||
| Model | Literal intent accuracy | NATO intent accuracy | Literal p50 | NATO p50 |
|
||||
|---|---:|---:|---:|---:|
|
||||
| `vosk-small-en-us-0.15` | 71.11% | 100.00% | 26.07 ms | 26.38 ms |
|
||||
| `vosk-en-us-0.22-lgraph` | 74.44% | 100.00% | 210.34 ms | 214.97 ms |
|
||||
|
||||
## Main Error Pattern (Literal Grammar)
|
||||
|
||||
- Letter confusion is concentrated on `p -> b`:
|
||||
- `control p -> control b`
|
||||
- `shift p -> shift b`
|
||||
- `control shift p -> control shift b`
|
||||
|
||||
## Takeaways
|
||||
|
||||
- NATO grammar is strongly validated for this keystroke use case (100% on both tested models).
|
||||
- `vosk-small-en-us-0.15` is the practical default for command-keystroke experiments because it matches NATO accuracy while being much faster.
|
||||
|
||||
## Raw Report
|
||||
|
||||
- `exploration/vosk/keystrokes/eval_runs/run-20260228T200047Z/summary.json`
|
||||
65
exploration/vosk/keystrokes/intents.json
Normal file
65
exploration/vosk/keystrokes/intents.json
Normal file
|
|
@ -0,0 +1,65 @@
|
|||
[
|
||||
{
|
||||
"intent_id": "ctrl+d",
|
||||
"literal_phrase": "control d",
|
||||
"nato_phrase": "control delta",
|
||||
"letter": "d",
|
||||
"modifier": "ctrl"
|
||||
},
|
||||
{
|
||||
"intent_id": "ctrl+b",
|
||||
"literal_phrase": "control b",
|
||||
"nato_phrase": "control bravo",
|
||||
"letter": "b",
|
||||
"modifier": "ctrl"
|
||||
},
|
||||
{
|
||||
"intent_id": "ctrl+p",
|
||||
"literal_phrase": "control p",
|
||||
"nato_phrase": "control papa",
|
||||
"letter": "p",
|
||||
"modifier": "ctrl"
|
||||
},
|
||||
{
|
||||
"intent_id": "shift+d",
|
||||
"literal_phrase": "shift d",
|
||||
"nato_phrase": "shift delta",
|
||||
"letter": "d",
|
||||
"modifier": "shift"
|
||||
},
|
||||
{
|
||||
"intent_id": "shift+b",
|
||||
"literal_phrase": "shift b",
|
||||
"nato_phrase": "shift bravo",
|
||||
"letter": "b",
|
||||
"modifier": "shift"
|
||||
},
|
||||
{
|
||||
"intent_id": "shift+p",
|
||||
"literal_phrase": "shift p",
|
||||
"nato_phrase": "shift papa",
|
||||
"letter": "p",
|
||||
"modifier": "shift"
|
||||
},
|
||||
{
|
||||
"intent_id": "ctrl+shift+d",
|
||||
"literal_phrase": "control shift d",
|
||||
"nato_phrase": "control shift delta",
|
||||
"letter": "d",
|
||||
"modifier": "ctrl+shift"
|
||||
},
|
||||
{
|
||||
"intent_id": "ctrl+shift+b",
|
||||
"literal_phrase": "control shift b",
|
||||
"nato_phrase": "control shift bravo",
|
||||
"letter": "b",
|
||||
"modifier": "ctrl+shift"
|
||||
},
|
||||
{
|
||||
"intent_id": "ctrl+shift+p",
|
||||
"literal_phrase": "control shift p",
|
||||
"nato_phrase": "control shift papa",
|
||||
"letter": "p",
|
||||
"modifier": "ctrl+shift"
|
||||
}
|
||||
]
|
||||
11
exploration/vosk/keystrokes/literal/phrases.txt
Normal file
11
exploration/vosk/keystrokes/literal/phrases.txt
Normal file
|
|
@ -0,0 +1,11 @@
|
|||
# Keystroke literal grammar labels.
|
||||
# One phrase per line.
|
||||
control d
|
||||
control b
|
||||
control p
|
||||
shift d
|
||||
shift b
|
||||
shift p
|
||||
control shift d
|
||||
control shift b
|
||||
control shift p
|
||||
10
exploration/vosk/keystrokes/models.example.json
Normal file
10
exploration/vosk/keystrokes/models.example.json
Normal file
|
|
@ -0,0 +1,10 @@
|
|||
[
|
||||
{
|
||||
"name": "vosk-small-en-us-0.15",
|
||||
"path": "/tmp/vosk-models/vosk-model-small-en-us-0.15"
|
||||
},
|
||||
{
|
||||
"name": "vosk-en-us-0.22-lgraph",
|
||||
"path": "/tmp/vosk-models/vosk-model-en-us-0.22-lgraph"
|
||||
}
|
||||
]
|
||||
11
exploration/vosk/keystrokes/nato/phrases.txt
Normal file
11
exploration/vosk/keystrokes/nato/phrases.txt
Normal file
|
|
@ -0,0 +1,11 @@
|
|||
# Keystroke NATO grammar labels.
|
||||
# One phrase per line.
|
||||
control delta
|
||||
control bravo
|
||||
control papa
|
||||
shift delta
|
||||
shift bravo
|
||||
shift papa
|
||||
control shift delta
|
||||
control shift bravo
|
||||
control shift papa
|
||||
Loading…
Add table
Add a link
Reference in a new issue