Multilingual Voice UX
Voice assistants routinely mishear accented English, and not randomly. This study found the specific, predictable acoustic collision points behind that and designed around them.
Voice assistants weren’t built for most of the world’s English speakers
Voice assistants are trained overwhelmingly on native American-English speech, and people with accents or non-native pronunciation regularly get misheard or misunderstood as a result. The usual response is to treat it as an unsolvable tolerance problem: accents are just hard for computers. This study started from a more useful question: what specifically breaks, acoustically, and is it actually fixable at the design level?
Three months, 22 speakers, and a classifier trained to listen for the gap
There was no separate engineer for this, so I built the classifier myself alongside running the study. I ran a longitudinal study with 17 non-native and 5 native English speakers, 22 total, recording vowel production three times over three months. I built a classifier comparing participants’ vowel formant frequencies against reference native-speaker datasets (Hillenbrand, and Assmann & Katz), then tested recognition accuracy under three background-noise levels to reflect real listening conditions instead of a clean lab signal.
87% accuracy, and a short list of exactly where it broke
The classifier predicted vowels correctly 87% of the time overall, solid for everyday use. The more useful finding was in the failures: they weren’t random.
Specific vowel sounds were reliably and repeatedly confused for each other. That reframed the whole problem. It wasn't a vague matter of "accents are hard"; it was a short, identifiable list of acoustic collision points.
Designing recovery paths around the specific failure points
Instead of treating the accuracy gap as something to route around generically, I used the specific failure points to inform user-flow design, building recovery paths exactly where the acoustic model was least reliable in place of one generic error state applied everywhere.
