We Measured Our Own Skills — Then Found Our Measurement Was Wrong

Most places that sell AI "skills" or prompts will tell you they work. Few will show you a controlled test of whether they actually do — and fewer still will publish it when the answer is no, or when their own test turns out to have been broken.

We've now done both. It's live at /benchmark, and this post is about what it took to get a number we actually trust.

The test itself

For each skill, we run the same task set twice: once with Claude given the skill's guidance, once with Claude given nothing but the raw task. An independent model — never the one that did the work — grades every output against a fixed rubric. Three runs per condition, to separate a real signal from ordinary run-to-run noise.

The number that sent us back to the drawing board

An early run of humanizer — a skill meant to make AI writing sound less robotic — came back with the skill scoring far worse than giving Claude nothing at all. Read at face value, that's a skill actively hurting the thing it's supposed to help.

We didn't publish that number and move on. We opened the actual transcripts, and found the test itself had real problems: the grader was scoring the skill's draft-and-notes scratch work instead of the final rewrite it handed back, several task prompts restated the skill's own instructions (so the no-skill arm had almost nothing to gain against), and a single pointwise judge call — no blind comparison, criteria always in the same order — carries known position bias. None of that is specific to humanizer; it was true of every skill in the catalog. A number that lopsided turned out to be a symptom of the harness, not (only) the skill.

What we changed

We rebuilt the grading engine before trusting any more numbers out of it:

• Grade the final deliverable, not the scratch work. Drafts and working notes are stripped before anything is scored.
• Blind, order-swapped pairwise comparison as the primary verdict: an independent judge sees both arms' output for the same task with no label saying which is which, in both orders — a side only wins if it's preferred both times, otherwise it's scored a tie, not a coin flip.
• Report headroom and cost, not just a pass-rate gap: how much room the no-skill baseline left to improve, and how many tokens the skill costs per output versus the plain model.
• Confidence intervals on every comparison, so a small task set doesn't get reported with false precision.

Then we re-ran the two pilot skills under the corrected engine.

What the corrected numbers actually show

humanizer — verdict: noise, not a loss. The blind pairwise comparison came back 12 wins, 16 losses, 2 ties for the skill (delta −3.3 points, 95% CI [−16.7, +6.7] — the interval straddles zero). That's genuinely inconclusive on today's task set, a different finding from "actively worse." The corrected engine also surfaced two concrete, fixable problems the old single number couldn't show: only 30% of the skill's outputs were paste-ready as returned (the rest needed trimming before use), and the skill costs about 15.6× the tokens of the plain model per output. Both are real, both are now visible, and neither is "the skill makes writing worse" — the deliverable itself reads more naturally with the skill on (80% vs. 47% on that specific check) than without it.

realtor_listing_copy — verdict: gain, and a clear one. The pairwise comparison came back 29 wins, 0 losses, 1 tie (delta +83.3 points, 95% CI [66.7, 96.7]). The clearest single driver: the skill's fair-housing-risk check passes 100% of the time with the skill versus 23% without — exactly the kind of failure mode a plain model has no reason to avoid on its own.

Both scorecards, with the full breakdown, are live on /benchmark right now.

Why we're publishing the process, not just the result

A benchmark you only show when it flatters you isn't a benchmark — it's marketing with a chart attached. That standard has to apply to our own measurement tooling too: when our first pass at grading produced a number that didn't hold up under inspection, the honest move was to say so and show the fix, not quietly swap in a better-looking number with no explanation. "The skill made things worse" was, on its own, still useful information worth publishing when we first saw it — it's a real, documented failure pattern (a companion piece: SkillsBench, an independent benchmark, found 16 of 87 tasks got measurably worse with a skill attached). What we're adding here is the harder, less flattering half: that number also has to be a correct measurement of what actually happened, and ours wasn't yet.

The rest of the catalog is still scored under the older, single-number method until each pack is re-run under the corrected engine — that re-baselining is ongoing, not instant, and a pack stays off /benchmark's published list until it's done under the current method rather than show a number we no longer fully trust.

What happens to a skill that scores like this

It doesn't get quietly deleted or have its numbers massaged. It's a candidate for our evolution loop — the same measurement engine gates any proposed rewrite before it ships, so a fix has to prove it's actually better, not just different, before it's accepted.

If you want to see whether a skill helps your specific use case rather than our test tasks, Measured Scorecards lets any Pro account run the same engine against their own opted-in packs — including, if you use Learning Mode, whether your own recorded corrections are actually moving the number.