Method note · AI in research
When an AI agent walks a menu, who decides if it found the thing?
I wanted to know whether AI agents could give researchers a first look at a navigation menu before anyone recruits a single participant. An agent gets a menu and a set of tasks, like "find where to pay a bill." It moves through one level at a time and writes down each step.
The part that matters is who decides whether the agent succeeded.
Version one: the agent reports its own result
In my first real sessions I let the agent say how each task went. It wrote things like "direct success" at the end of a task. Then a researcher checked the recorded path against the menu.
Close to half of the agent's own outcome claims were wrong. It would report success on a task where its path ended somewhere else entirely. Nothing in the agent was lying. A model that has just walked a path tends to describe that path as reasonable, so its account of what happened and what happened drift apart.
Version two: a second AI scores the first
The obvious fix was a second agent to score the first one's record. That helped less than I expected. The scorer never saw the menu, only the record, so when the first agent invented a menu item, the scorer treated it as real and penalized the menu for it. Two AI steps in a row passed the same mistake along.
Version three: code answers the factual question
The question "did the path end at the right place" has one correct answer. It isn't a judgment call, so it doesn't need a judge. In the third round I moved it into code. The agent records its steps. Code compares the last step to the target and works out the outcome. Every score is recalculated from the recorded steps each time, so nothing stale survives a correction.
The agent still does what it is good at: reading labels and describing what it would try next. A researcher still makes the real judgment calls, such as whether a label is unclear or a task has two valid routes.
Even then, the agent over-reported success. Most of the failures it called successes were caught only because code, not the agent, held the answer.
What I'd take from it
- Match the tool to the shape of the question. Factual outcomes go to code. Descriptions of reasoning go to the agent. Judgment calls go to a person.
- Make every claim carry its evidence. If a finding can't point to the step that produced it, it doesn't go in.
- Run each setup more than once. One run is an anecdote.
- Read the reasoning, not just the score. In my first round the agents' reasoning showed confusion on tasks that scored as clean successes. The scores alone understated the effort.
An agent walking a menu is not a customer. It can show what a menu's wording and structure do to someone moving through it, and it can surface obvious problems early. It can't predict what real people will do, and I never use it that way. Its job is to make the test with real participants spend its time on the questions only people can answer.
I call the project Poindexter. The full study, with the sprint-by-sprint detail, is a case study I share with people I'm talking to. See what it covers or request access.