Research Signal Pipeline

An intent field that was wrong nearly half the time

Verizon’s digital properties collect tens of thousands of intercept survey responses every month, and each one is supposed to say why the customer showed up. The field that was meant to capture that labeled roughly half of all sessions with the same catch-all category, regardless of what the customer actually came to do. That’s not a weak signal, it’s close to no signal at all, and it made a huge share of downstream research unreliable by default. Friction analysis was effectively blind: there was no trustworthy way to tell whether someone struggling to pay a bill was hitting different friction than someone trying to troubleshoot a device, because the data couldn’t reliably tell those customers apart in the first place.

There was no separate engineering track to rebuild this. As the researcher who depended on the data, I built the pipeline myself rather than wait for it to be prioritized elsewhere.

A two-stage pipeline, built like a research instrument

The system runs in two connected stages, each with its own name since each has its own job. FelixAI extracts a plain-language friction summary from each session’s raw activity, under a strict output contract so results stay usable at scale instead of turning into inconsistent prose. Intent Classifier takes that summary as its strongest input and classifies each session into an intent taxonomy for standard survey respondents who self-report why they visited; a sibling model, Classifier v2, runs the same taxonomy against a much larger population that never answers that question at all.

Getting the intent taxonomy right took three attempts. The first pass was too broad, with a single catch-all category absorbing nearly a third of all sessions, no more useful than the field it replaced. The second pass overcorrected into categories so granular that several had almost no examples to learn from and confidence scores that never became trustworthy. The version that shipped was calibrated against the actual distribution of the data, with a rule that no category could go live without a meaningful minimum of labeled examples behind it. That rule, more than any single model choice, is what made the taxonomy stable.

Intent Classifier was validated against over 15,000 sessions labeled by hand against a written rubric, with a portion independently spot-checked. Every classification below a set confidence threshold gets flagged rather than trusted outright, the same instinct that governs when a qualitative finding needs another data point before it goes in a report.

The strongest signal wasn’t always available, and that revealed something else

1. 01
   Accuracy more than kept pace with scale.

   Classification accuracy moved from roughly 55% under the old field to 87.2% across 183,841 processed sessions, validated against the held-out labeled set rather than estimated.

2. 02
   Losing the strongest signal cost real accuracy, but most of it was recoverable.

   The single most reliable input is a customer's own self-reported reason for visiting. Sessions without that self-report lost several points of accuracy, recovered mostly by falling back through a prioritized cascade of weaker signals instead of guessing blind.

3. 03
   Fabricated detail was a bigger risk than missing detail.

   Early versions of FelixAI invented plausible-sounding technical error details that couldn't be verified against real system data. Constraining the model to only report errors it could match against a known, approved set, and to say "unknown" otherwise, cut that fabrication rate from roughly 31% to under 2% over four prompt revisions.

Building the extraction step surfaced edge cases that don’t show up until real data breaks them:

• A session with no usable summary at all, rather than a wrong one
• Corrupted or partial session data
• Sessions with no discernible friction to report
• Multiple, roughly equal candidate frictions in the same session

Each got an explicit rule rather than being left to the model’s judgment, the same way a good research protocol writes down what to do with an ambiguous participant response instead of deciding in the moment.

The segment nobody could see under the old label

Filtering the new dataset for connectivity-related friction surfaced customers looping through the same troubleshooting steps for several pages before giving up entirely, a pattern that was completely invisible under the old catch-all label. That finding fed directly into a product team’s prioritization conversation. More broadly, FelixAI and Intent Classifier together turned an unreliable field into a structured, validated dataset spanning 183,841 sessions and roughly 35,000 new responses every month, now the foundation other research on the site can build on with confidence in what it’s actually measuring.

It’s also a proof point for the shape of research I want to keep doing: applying the same rigor I’d bring to a study design, labeled validation sets, calibration rules, defined failure modes, directly to building the systems that make research possible in the first place.