Measurements, not promises
Last updated: September 2026
VAYS runs its AI on your device by default. Claims like that are easy to make and hard to check, so this page shows how we test: real benchmarks, run on the exact builds we ship, with the caveats spelled out.
Polish medical speech: the hardest test we could find
ADMEDVOICE is a public corpus of Polish medical dictation published in Nature Scientific Data (2025, DOI 10.1038/s41597-025-05776-1, CC BY 4.0): radiology reports, oncology notes, referrals, resuscitation records. Medical speech is a worst case for any recognizer: specialist vocabulary, drug names, dosages and units.
We measured the exact speech stack VAYS runs on-device (NVIDIA Parakeet v3, fully offline via CoreML on Apple Silicon) on the natural part of the corpus. The numbers below come from the updated engine now live in iOS 1.9 and Mac 2.8. To our knowledge this is the first published measurement of this engine on Polish medical speech.
| Metric | VAYS on-device stack |
|---|---|
| Recordings | 9,797 |
| Audio | 16.3 h |
| Word error rate (WER) | 21.4% |
| WER, numbers normalized | 20.1% |
| Character error rate (CER) | 9.4% |
| Speed on an Apple Silicon Mac | 65x faster than realtime |
Difficulty varies a lot by specialty: referrals and recommendations land near 10% WER, while cardiology reaches 44% (dosages, parameters, units). Part of that tail is the corpus itself: we identified 80 recordings with mismatched audio and reference text in the source data; with those filtered out, cardiology improves to about 37%.
Reference points
For context, results reported on Polish medical speech by others. The basis differs. Our 21.4% is measured on ADMEDVOICE, a public corpus of single-speaker dictation, while the arXiv:2603.02246 numbers come from that team's own corpus, more than 9 hours of doctor and patient dialogue. Their two figures are also number-normalized, where the 21.4% here is our raw WER; with numbers normalized ours is 20.1%. Different corpora, different kind of speech, different metric, so treat this as orientation, not a ranking.
| System | WER | Notes |
|---|---|---|
| VAYS (on-device, this measurement) | 21.4% | Runs offline on your Mac or iPhone |
| Whisper-medium fine-tuned by the corpus authors | 9.0% | Our measurement of their model on our sample; it saw part of this data in training, treat as a best case |
| ElevenLabs Scribe | 10.6% | Cloud service, audio leaves the device. Number from arXiv:2603.02246, measured on that team's own dialogue corpus |
| Whisper | 42.3% | Baseline in the same paper, on the same corpus; the paper does not state which Whisper variant it ran |
The engine keeps improving
In August 2026 we updated the speech engine and re-ran the full benchmark before shipping. Accuracy held (WER 21.43% vs 21.47%), speed improved by 28%, and one bug class disappeared completely: short Polish utterances occasionally came back with stray Cyrillic characters. That happened 22 times across the corpus on the old engine and zero times on the new one.
Method and honesty
- We measure the real CoreML stack on Apple hardware, not a reference GPU build. The speed figure comes from one Apple Silicon configuration; older machines will be slower.
- ADMEDVOICE is single-speaker dictation. Conversational audio, meetings and noisy rooms will score differently.
- WER counts every inserted, missing or altered word. A 21% WER text is imperfect but very usable as a draft, which is how VAYS treats transcripts: as material you review, not as ground truth.
- The corpus is public (CC BY 4.0). Protocol details and our measurement scripts are available on request.
Interested in what this means for medical documentation? Read about VAYS MED.