Research Integrity Toolkit
Three tools built to keep AI-assisted research defensible: bias-checked survey instruments, an evaluation architecture with traceable evidence, and one consistent research vocabulary.
Research methodology is easy to get right in theory and wrong under deadline
A leading survey question slips through when a launch is close. An AI-generated evaluation sounds confident and nobody can check where the confidence came from. Terminology drifts across thousands of reports until nobody agrees what a term even means anymore. None of these are new problems, but AI-assisted research adds a new version of each: a model can produce a biased question, a plausible-sounding but ungrounded evaluation, or an inconsistent label just as easily as a rushed human can, and often faster. Rather than one large tool, I built three small ones, each aimed at a different point where trust in research output actually breaks down.
Three tools, three different failure modes
Survey Smith checks survey instruments before they go out. It encodes methodology knowledge, the kind normally applied by feel, as versioned, testable rules rather than tacit judgment: a double-barreled question asks two things at once and overloads working memory, leading language activates confirmation bias, a missing neutral option forces a false choice. Each rule carries its own severity and rewrite guidance, and the whole set is validated against a labeled corpus of real and synthetic survey questions rather than trusted on intuition alone.
Project Poindexter addresses a subtler problem: single-agent AI evaluation is contaminated by design, because an observation and the conclusion drawn from it happen in the same context, and the conclusion quietly shapes what gets noticed. Human research protocols solve this by separating observation from analysis. I built the same separation into a dual-agent architecture instead of assuming a single careful prompt would hold the line.
An Observer agent describes an artifact under strict constraints, no interpretation, no evaluation language, and produces a sealed record before anything is scored. An Evaluator agent then scores that record against defined criteria, citing specific passages as evidence, without ever seeing the original artifact again. Two tools run on Project Poindexter’s architecture: Tree Test, an information-architecture auditor that scores findability and mislabeled navigation, and Heuristic Eval, built on Nielsen’s ten usability heuristics, each scored independently with cited evidence.
Research Doc Processor is smaller and more mechanical: a two-stage pipeline that reads a large archive of historical research reports, extracts every acronym and synonym cluster, and merges them into one deduplicated reference instead of leaving that knowledge scattered across thousands of individual documents.
Eleven findings you can stand behind beat fourteen you can’t
Survey Smith validated at 84% precision and 80% recall across a 140-question labeled corpus, weighted across categories, with the tool deliberately tuned to flag only above a real confidence threshold rather than over-flag and train researchers to start ignoring it.
- 01Separating observation from evaluation produced fewer, more defensible findings.
Run on the same artifact, a single-agent baseline produced 14 findings but only 43% carried evidence that survived review. Project Poindexter produced 11 findings, every single one traceable to a cited passage in the sealed record, with zero rated unsupported.
- 02Severity inflation disappeared along with the unsupported findings.
The single-agent baseline showed a pattern of severity creep, findings rated more serious than the evidence actually supported. That pattern was absent once observation and evaluation were structurally separated, not just prompted to be careful.
- 03Research Doc Processor replaced roughly 250 hours of manual work for about $22 in model cost.
Extraction ran on a fast, inexpensive model since recognizing an acronym is a lookup task; deduplication and conflict resolution ran on a more capable model since deciding whether two terms mean the same thing is a judgment task. Splitting the two by difficulty, not defaulting to the most powerful model everywhere, was most of the cost savings.
Project Poindexter has a real limitation: it's only as good as the observer's description, and if the observation record misses something, the evaluator has no way to find it, the same limitation a human blind evaluation has. Naming that plainly, rather than treating it as a flaw to hide, is itself part of the methodology. A research tool that's honest about what it can't catch is more trustworthy than one that quietly claims full coverage.
The same rigor applied to the tools, not just what they produce
Survey Smith, Project Poindexter, and Research Doc Processor share one instinct: treat an AI system the way a research protocol treats a human observer, with defined procedures, stated confidence levels, and honest limitations, rather than trusting output because it sounds fluent. Together with FelixAI and Intent Classifier on the signal side, they form one practice rather than a set of disconnected experiments: build the instruments carefully, evaluate them so the evidence is checkable, and keep the vocabulary consistent enough that the research stays usable at scale.
Bias-detection precision by category
Double-barreled questions
Leading language
Loaded framing
Social desirability