Plx Alerts Usability Study
A baseline usability study on Plx Alerts, Google's internal alerting tool, that pinpointed exactly which form fields were costing engineers the most time and confidence.
An alerting tool nobody had baselined before scaling it up
Plx Alerts let engineers, team leads, and program managers set up automated monitoring on their data, so they’d hear about a problem before a customer did. It was heading toward general availability, but nobody had measured its actual usability first, only whether it technically worked. Before it scaled to more teams, the question was simple: could someone unfamiliar with the tool actually get through the core jobs it was built for, and where exactly would they get stuck?
Ten engineers, four real tasks, one shared rubric
I ran a 70-minute moderated lab session with 10 participants (5 engineers, 2 team leads, a technical program manager, and a partner technology manager), recruited from active Plx Alerts and Plx Dashboards user groups. Each completed four tasks under a time limit agreed on with the product team in advance: creating a new rule, modifying an existing alert, resolving an alert, and bulk-resolving alerts. Every session was transcribed into a sequence model and coded using open and axial coding to find where people broke down, not just whether they finished.
The form worked. Explaining itself to a new user didn’t.
- 01The configuration form assumed knowledge nobody had yet.
9 of 10 participants said the form gave them insufficient information to fill in or edit a rule. The Time field alone confused 8 of 10; one participant asked outright whether it meant a timestamp or a duration, with nothing in the interface to tell them.
- 02The preview couldn't be trusted.
9 of 10 weren't confident a new rule would actually monitor the right data slice once saved, and the preview interface gave them no way to verify it before committing. High load times (8 of 10) discouraged people from experimenting further to check their own work.
- 03Novices had no on-ramp.
8 of 10 wanted more guidance picking a detection method, and 6 of 10 said building, debugging, or tweaking a rule as a first-timer was genuinely complex. Once past that hump, though, 3 of 10 said Plx Alerts compared favorably to other monitoring tools they'd used.
Despite the friction, participants liked the tool once they got past the learning curve: "I definitely like the new UI. It's way better than the second generation one, I would say it's more cleaner." Three participants clicked the thumbs-up button on the explanation window unprompted during their session. The tool's ceiling was fine; the climb to get there was the actual problem.
Testing also isolated exactly which fields and labels were the actual friction points, specific enough to hand straight to design:
- What the "Time" field expects as input8 of 10
- Whether "Autobreakdown" is needed for monitoring slices6 of 10
- What option to pick in the "Granularity" field5 of 10
- What functions are valid in the "Measures" field4 of 10
A prioritized fix list instead of a general “make it easier” ask
Recommendations mapped directly onto what broke: instruction bubbles next to the Time, Granularity, and Measures fields, inline validation with real error messages instead of silent failures, and a confirmation message once a rule was actually saved. One fix, a loading spinner for the preview interface, was already implemented by the time the report shipped. The study gave the team a baseline SUS of 73.5 to measure every future iteration against, the same benchmarking approach later used on Query Profiler 2.0.