← All work

Verizon · Digital Channels20263 min readAI

Testing Verizon's Navigation With AI Agents, and Checking Their Work

I built a way to use AI agents for a first look at Verizon's navigation menus, then found the agents rarely admitted when they failed. Code now checks their work, and a researcher makes the calls.

57 of 68
Failed tasks that the AI agent called a success
98
AI navigation sessions across three sprints
37% vs 81%
How often the AI's own account was right: app menu vs website account area

Role: Lead researcher, designed the instrument

  • Study design
  • Information architecture
  • Instrument design
  • AI evaluation
  • Mixed methods
01Context

A fast first look at a menu, before any customer sees it

Before a navigation menu goes in front of customers, researchers want a first read on whether people will find things in it. A real tree test with participants takes recruiting and time. AI agents can walk a menu in seconds, so I wanted to know whether they could give a first look that a researcher could stand behind.

The agents can show what a menu’s wording and layout do to someone moving through it. They can’t predict what customers will do, and the instrument never claims to.

I designed the study and the instrument. Two other researchers helped me run the experiments. I called the project Poindexter.

02Method

How I built it, and how it kept breaking

The setup is simple. An AI agent, the navigator, gets a menu and six tasks, like finding a plan or paying a bill. It moves through the menu one level at a time and writes down every step. Then the record is scored, and a researcher reviews whatever gets flagged.

Exhibit A: Who checks whatEach job goes to the right tool
AIWalks the menu and writes down each step
CODEWorks out whether the path ended in the right place
CODEScores the record from the recorded steps
RESEARCHERReviews what gets flagged and makes the calls

It didn’t start that way. It took three sprints, and each one ended with a list of things that broke.

Exhibit B: Three sprints, each one rebuilding the instrument
Sprint 1Mar 31 · 18 sessions · 6 prompt setups
What broke
  • Agents invented menu items and whole paths
  • The same label path scored 34, 83, and 95 for readability across three researchers
  • None of the 18 sessions noticed a task with three valid starting points
What changed next
  • Agents may only read the items in front of them
  • Retired the readability formula for a labeling scheme built for menus
  • Researcher review became mandatory
Sprint 2May 11 to 15 · 24 sessions · 144 tasks · 3 AI models
What broke
  • 47% of the AI's own outcome claims were wrong (68 of 144)
  • One model reported a top-level menu item that doesn't exist, in all 48 of its tasks
  • In two cases the same setup scored 42 and 56 points apart across researchers
What changed next
  • Outcomes computed by code, not claimed by the AI
  • The AI scorer retired, scores recalculated from the recorded steps
  • Every setup run three times
Sprint 3May 27 to Jun 9 · 56 sessions · 336 tasks · 2 AI models · 3 menus
What broke
  • The AI still overclaimed: 57 of 68 failures called successes
  • Only 31 of 56 sessions were clean enough to trust
  • One task with two valid routes was scored a failure
What comes next
  • Run both models side by side on the same menus
  • Settle the two-route question before reporting it

The biggest change was about who gets to judge. In the second sprint a second AI scored the first one’s record, and it inherited the first one’s mistakes: it charged penalties for a menu item the navigator had invented. The navigator’s own account of what it did couldn’t be trusted either. So in the third sprint I took the outcome out of the AI’s hands. Code works out whether the path ended in the right place, every score is recalculated from the recorded steps, and every judgment has to carry its evidence.

The AI still does what it’s good at, walking a menu and reading labels. The factual question gets answered by code, and the judgment calls stay with a researcher.

I also read the agents’ reasoning, not only their scores. In the first sprint the reasoning logs showed confusion on tasks that had scored a clean success, so the scores alone were understating the effort.

03Findings

What 336 tasks showed

In the third sprint, two other researchers and I ran 56 sessions on three Verizon menus with two AI models. That’s 336 tasks.

Exhibit C: What the AI said about the 68 tasks that actually failedOut of 336 tasks
  • 57called it a success
  • 8left blank
  • 3admitted failure

A task counted as failed when the recorded path ended somewhere other than the correct destination. 53 of the 68 were confident trips to the wrong place.

  1. 01
    The AI almost never admitted failure.

    Of the 68 tasks that failed, the navigators called 57 a success, admitted 3, and left 8 blank.

  2. 02
    On the hardest menu, its own account was close to useless.

    On the app menu, the AI’s description of its outcome was right 37% of the time, and 34% of tasks failed. On the website’s account area it was right 81% of the time, with 15% failing.

Exhibit D: How often the AI's own outcome was rightShare of tasks where its claim matched what happened
81%
Website account area
69%
Global menu
37%
App menu

114, 108, and 114 tasks per menu. The three menus differ in structure and depth, so this shows where the AI's account broke down and not why.

  1. 03
    Running the same thing twice gave different answers.

    Repeating the same setup moved the 100-point health score by a median of 18 points, and by more than 20 in 6 of 18 setups. A single run isn’t evidence, so every setup ran three times.

  2. 04
    Only 31 of 56 sessions were clean enough to trust.

    A session had to follow the rules at least 80% of the time, with no compromised tasks and nothing unresolved. Claude cleared that on 19 of 29 sessions and Gemini on 12 of 27. Claude’s tasks failed 14% of the time, Gemini’s 27%.

The method works as a menu audit too. In every clean session the menus had too many choices at one level, and in 27 of 31 some labels were unclear.

What I’d tell anyone using AI agents this way

Don’t take an agent’s word for how it did. Compare what it says with what actually happened, and let something it can’t edit decide.

04Outcome

What’s still unsettled

The third sprint is finished, but I haven’t formally closed it. The retrospective isn’t written yet.

One open question matters more than the rest. One app task has two valid routes. Gemini kept taking the second one and was scored a failure for reaching a valid place. Whether that’s a flaw in my instrument or a finding about the menu isn’t decided, and I won’t show that task’s results to the design team until it is. Next I’ll run both models side by side on the same menus, and settle that question first.

Anyone can rerun a finding from the third sprint. The recorded session goes through the same code and comes out with the same score.

Limits worth knowing

No real participants took part, so every finding is about the structure of the menu and none predicts what customers will do. There’s no human baseline to compare the agents against, and one of the three models I planned to test lost access before the third sprint. I didn’t pre-register my success criteria.