Query Profiler 2.0

A legacy tool everyone needed and nobody enjoyed using

Query Profiler was critical infrastructure for Google’s Core Data org. Engineers, data scientists, and PMs used it daily to analyze and optimize query performance, but it ran on outdated technology, carried a third-party dependency, and had a usability problem nobody had fully quantified. A 2017 study put a number on it: a System Usability Scale score of 43.7, solidly in “not acceptable” territory, with only 53% task success. A follow-up study in 2019 found people couldn’t reliably tell how the tool’s “fragments” connected to each other, wanted clearer terminology, and leaned on a plan graph most of them didn’t fully trust. Query Profiler 2.0 was already being rebuilt to fix this. My job was to confirm the rebuild actually did, before it shipped to every engineer at the company.

Two-phase testing, gated by real launch criteria

I ran a two-phase iterative program instead of a single validation study. Phase I (formative) used small groups of 4-6 participants in 30-45 minute think-aloud sessions, catching problems early while the design could still change easily. Phase II (summative) used 10-12 participants in 60-minute retrospective think-aloud sessions plus a standardized SUS questionnaire, run once the design was stable enough to serve as a genuine launch-readiness gate. Participants were recruited and segmented deliberately: by usage frequency (power users, regular users, and people who’d used the tool once or never) and by role (software engineers, data scientists, PMs, tech leads). Every test task was pre-classified into features critical for launch versus merely nice to have, so the team always knew exactly what had to work before ship.

Usability improved fast. Confidence took longer to catch up.

1. 01
   Real, uneven progress.

   Usability climbed from a 48.9 baseline to 81.2 by the final round, though iteration 2 dipped before recovering. That's the ordinary shape of real iterative testing.

2. 02
   It surfaced problems better than it resolved them.

   Error messages successfully helped participants identify the source of an issue (10 of 12), but most remained unsure how to actually fix it (10 of 12) or whether their fix had worked at all (9 of 12).

3. 03
   Confidence lagged behind unfamiliarity with the redesign.

   9 of 12 participants said they needed more time exploring the new interface before feeling comfortable. Several specific fields (Time, Granularity, Auto Breakdown) were still unclear by name alone.

Testing also pinpointed exactly which field labels were causing confusion, specific enough to act on directly:

• What the "Time" field expects as input8 of 10
• What "Auto Breakdown" does and when to use it6 of 10
• What option to choose in the "Granularity" field5 of 10
• Which functions are valid in the "Measures" field4 of 10

From a usability gate to a template for how the team ships

The recommendations followed the findings directly: interactive in-tool onboarding instead of a static help page, contextual hints linking straight to relevant documentation at the fields that tested unclear, and a richer fragment pop-up explaining why a query was being throttled. Post-launch, satisfaction climbed from 43% to 76% over three months, and the team projected a 2.2-point lift in engineering productivity alongside an 18% reduction in operational costs within a year. The project also became a template for user-centered launch gating that the org reused afterward, and it contributed directly to the research team’s own growth.