Verizon · Digital Channels20263 min readAI
Testing Verizon's Navigation With AI Agents, and Checking Their Work
I built a way to use AI agents for a first look at Verizon's navigation menus, then found the agents rarely admitted when they failed. Code now checks their work, and a researcher makes the calls.
- 57 of 68
- Failed tasks that the AI agent called a success
- 98
- AI navigation sessions across three sprints
- 37% vs 81%
- How often the AI's own account was right: app menu vs website account area
Role: Lead researcher, designed the instrument
A fast first look at a menu, before any customer sees it
Before a navigation menu goes in front of customers, researchers want a first read on whether people will find things in it. A real tree test with participants takes recruiting and time. AI agents can walk a menu in seconds, so I wanted to know whether they could give a first look that a researcher could stand behind.
The agents can show what a menu’s wording and layout do to someone moving through it. They can’t predict what customers will do, and the instrument never claims to.
I designed the study and the instrument. Two other researchers helped me run the experiments. I called the project Poindexter.
How I built it, and how it kept breaking
The setup is simple. An AI agent, the navigator, gets a menu and six tasks, like finding a plan or paying a bill. It moves through the menu one level at a time and writes down every step. Then the record is scored, and a researcher reviews whatever gets flagged.
It didn’t start that way. It took three sprints, and each one ended with a list of things that broke.
- Agents invented menu items and whole paths
- The same label path scored 34, 83, and 95 for readability across three researchers
- None of the 18 sessions noticed a task with three valid starting points
- Agents may only read the items in front of them
- Retired the readability formula for a labeling scheme built for menus
- Researcher review became mandatory
- 47% of the AI's own outcome claims were wrong (68 of 144)
- One model reported a top-level menu item that doesn't exist, in all 48 of its tasks
- In two cases the same setup scored 42 and 56 points apart across researchers
- Outcomes computed by code, not claimed by the AI
- The AI scorer retired, scores recalculated from the recorded steps
- Every setup run three times
- The AI still overclaimed: 57 of 68 failures called successes
- Only 31 of 56 sessions were clean enough to trust
- One task with two valid routes was scored a failure
- Run both models side by side on the same menus
- Settle the two-route question before reporting it
The biggest change was about who gets to judge. In the second sprint a second AI scored the first one’s record, and it inherited the first one’s mistakes: it charged penalties for a menu item the navigator had invented. The navigator’s own account of what it did couldn’t be trusted either. So in the third sprint I took the outcome out of the AI’s hands. Code works out whether the path ended in the right place, every score is recalculated from the recorded steps, and every judgment has to carry its evidence.
The AI still does what it’s good at, walking a menu and reading labels. The factual question gets answered by code, and the judgment calls stay with a researcher.
I also read the agents’ reasoning, not only their scores. In the first sprint the reasoning logs showed confusion on tasks that had scored a clean success, so the scores alone were understating the effort.
What 336 tasks showed
In the third sprint, two other researchers and I ran 56 sessions on three Verizon menus with two AI models. That’s 336 tasks.
- 57called it a success
- 8left blank
- 3admitted failure
A task counted as failed when the recorded path ended somewhere other than the correct destination. 53 of the 68 were confident trips to the wrong place.
- 01The AI almost never admitted failure.
Of the 68 tasks that failed, the navigators called 57 a success, admitted 3, and left 8 blank.
- 02On the hardest menu, its own account was close to useless.
On the app menu, the AI’s description of its outcome was right 37% of the time, and 34% of tasks failed. On the website’s account area it was right 81% of the time, with 15% failing.
114, 108, and 114 tasks per menu. The three menus differ in structure and depth, so this shows where the AI's account broke down and not why.
- 03Running the same thing twice gave different answers.
Repeating the same setup moved the 100-point health score by a median of 18 points, and by more than 20 in 6 of 18 setups. A single run isn’t evidence, so every setup ran three times.
- 04Only 31 of 56 sessions were clean enough to trust.
A session had to follow the rules at least 80% of the time, with no compromised tasks and nothing unresolved. Claude cleared that on 19 of 29 sessions and Gemini on 12 of 27. Claude’s tasks failed 14% of the time, Gemini’s 27%.
The method works as a menu audit too. In every clean session the menus had too many choices at one level, and in 27 of 31 some labels were unclear.
Don’t take an agent’s word for how it did. Compare what it says with what actually happened, and let something it can’t edit decide.
What’s still unsettled
The third sprint is finished, but I haven’t formally closed it. The retrospective isn’t written yet.
One open question matters more than the rest. One app task has two valid routes. Gemini kept taking the second one and was scored a failure for reaching a valid place. Whether that’s a flaw in my instrument or a finding about the menu isn’t decided, and I won’t show that task’s results to the design team until it is. Next I’ll run both models side by side on the same menus, and settle that question first.
Anyone can rerun a finding from the third sprint. The recorded session goes through the same code and comes out with the same score.
No real participants took part, so every finding is about the structure of the menu and none predicts what customers will do. There’s no human baseline to compare the agents against, and one of the three models I planned to test lost access before the third sprint. I didn’t pre-register my success criteria.