Documentation Metrics Program
A recurring survey program that measured documentation quality across a dozen internal engineering tools, then tracked whether fixes actually moved the needle.
Documentation quality was managed by instinct instead of evidence
Across roughly a dozen internal engineering tools, nobody had a reliable way to tell which tool’s documentation was actually causing the most confusion, or whether editing a doc after a complaint had helped at all. Complaints came in one at a time, anecdotally, with no baseline to compare against and no way to measure whether a fix had worked.
A lightweight survey that measured five dimensions on a rotation
Working with the technical writing team, I built a recurring in-product survey using the HaTS (Happiness Tracking Survey) format: three questions, shown to real documentation users, rotating through five dimensions, discoverability, coverage, clarity, freshness, and organization, one at a time. To avoid survey fatigue, each prompt waited 30 seconds and 4 page views before appearing, then didn’t reappear for 90 days after a response. Every quarter, open-text comments were coded thematically to find recurring patterns.
People didn’t want more documentation. They wanted it to actually run.
- 01Coverage complaints had one clear ask.
6 of 12 comments on coverage asked for complete, runnable examples instead of isolated code snippets; 4 of 12 said the snippets alone weren't helpful at all.
- 02Discoverability and organization had the most room to improve.
One participant flagged a specific internal doc as too large to navigate and suggested splitting it into smaller, topic-specific pages, a fix concrete enough to act on directly.
- 03Scale varied enormously by tool.
Over the same 13-month window, the two most-used tools drew very different volumes, 1,345 responses for one internal data tool versus 636 for another, meaning a documentation fix to either one reached a large, recurring audience of engineers.
Not every fix worked, and the program was built to catch that honestly: after a documentation edit shipped on one tool, a follow-up read showed no statistically significant change in ratings. Knowing a fix hadn't moved the needle was as useful as knowing one had, since it redirected effort toward the fixes that actually mattered.
From one-off survey to a standing quarterly practice
Recommendations turned into concrete deliverables: cataloged commands in cheat-sheet form, examples tied to a user’s actual goal rather than generic snippets, and a specific call to make a real structural change to navigation and organization, then measure it properly instead of tweaking incrementally and guessing. The program itself outlived any single quarter’s findings; the technical writing team adopted it as a standing measurement tool for tracking documentation health over time.