What if we could observe how AI agents follow agentic skills? We can. Spawn another agent process in background, read its output, score against ground truth. TDD or User Testing for skill files.

I was updating our serpapi-web-search skill and wanted to know: does this specific documentation help LLM agents correctly solve the task?

Results

Fixes

This approach helped self-fixing several documentation issues:

Test runs

Example prompt: "How many Google Maps reviews does Tartine Bakery have?"
Ground truth: 5,915.

With the skill: agent calls SerpApi, gets 5,915.
Without: Opus refused, Fable launched Playwright (works 5/12 times), GPT guessed confidently wrong.

For the different task, GPT gets exact flight prices WITHOUT the skill ($299, matching API answer, 5/6 correct). Fable and Opus: 0/12.

We ran 432 of these.

Task With Without Notes
Scholar citations100%28%Without gives stale 265K vs exact 257K
Flight prices100%14%GPT gets it via web search; Fable/Opus can't
Maps reviews100%19%Fable uses Playwright; Opus refuses; GPT guesses
Hotel prices100%0%Refuses: "I can't look up live hotel prices"
Image source100%0%Hedges or guesses wrong domain
Product price100%33%Gives $449-499 estimates; actual $649
Shopping price67%0%1 auth failure; gives ~$198 vs $248 actual
YouTube views100%92%Zero lift. Web search returns exact counts
App Store rating100%100%Zero lift. Public API endpoint
JWST mass97%97%Control. Training data
News headlines100%100%Control. Any web search works

Scholar + Flights + Maps: WITH 100%, WITHOUT 20%. Haiku provided correct answers only when skill was provided.

How it works

The AUT (Agent Usability Testing) skill runs on two conditions:

  1. WITH: skill file loaded + tool available
  2. WITHOUT: no skill file, no tool

WITH scores higher: the skill was useful. WITHOUT scores same: it wasn't useful.

It guides an agent through the steps:

  1. Write tasks your tool should help with (and control tasks it shouldn't)
  2. Run WITHOUT. Watch it fail. That's your red test.
  3. Run WITH. Watch it pass. That's green.
  4. Edit one line of docs, rerun. Did it break? That's refactor.
  5. Repeat across models.

Lessons

I did several mistakes during evaluation:

  1. Compared real $HOME (all configs and extensions) vs stripped HOME. Not a fair comparison.
  2. Running from the repo directory. Agent grepped the working dir and found skill files.
  3. A second serpapi binary in ~/.cargo/bin survived PATH shadowing.

Agents didn't auto-discover tools

Agents didn't auto-discover tools (thanks Cap)

I ran a third condition: tool on PATH and authenticated, but no skill file.

0 out of 24 runs. Never typed serpapi. Never explored PATH. Never discovered it.

Conclusion

We tested this "TDD for skills" approach with Copilot CLI and SerpApi and fixed several documentation issues, that the /review on clean context didn't notice. Please give the it try and share your results.

P.S. For transparency, below are all tasks during the tests:

Phase 1: Easy tasks (126 runs, all 100%/100%, zero lift)

  • "What is the phone number for Tartine Bakery in San Francisco?"
  • "What is Apple's current stock price?"
  • "What is the price of the iPhone 15 Pro?"
  • Other simple lookup questions that built-in web_search handles trivially

Phase 2: First hard tasks (90 runs, N=3)

  • "Exact citation count for 'Attention Is All You Need' on Google Scholar"
  • "Cheapest nonstop flight JFK→LAX tomorrow, one-way"
  • "Price of DeWalt DCD771C2 drill on Google Shopping by retailer"
  • "Number of Google Maps reviews for Tartine Bakery, SF"
  • "Google Maps rating for Tartine Bakery, SF"

Phase 3: Expanded 240-run retest (10 tasks, 6 engines)

  • Scholar: AlexNet citations, BERT citations
  • Flights: JFK→SFO one-way, SFO→NRT
  • Shopping: DeWalt drill price
  • Maps: Central Park review count, Tsukiji rating
  • Hotels: cheapest Kyoto hotel on specific date
  • App Store: Notion app rating

Phase 4: Final N=12 matrix (432 runs, 3 models)

  • "How many citations does 'Attention Is All You Need' have on Google Scholar?"
  • "What is the cheapest nonstop flight from JFK to SFO on 2026-07-09, one-way?"
  • "How many views does the official Rick Astley Never Gonna Give You Up video have on YouTube?"
  • "How many Google Maps reviews does Tartine Bakery in San Francisco have?"
  • "What is the mass of the James Webb Space Telescope's primary mirror in kilograms?"
  • "What are the top 3 technology news headlines right now?"

Phase 5: Expanded engines (36 runs, Opus only, N=3)

  • "What is the price of the PlayStation 5 Disc Console Slim on Walmart?"
  • "What is the price of the cheapest Nintendo Switch OLED listing on eBay?" (invalid - no stable ground truth)
  • "What is the all-time rating of the Notion app on the Apple App Store?"
  • "What is the cheapest hotel in Kyoto for the night of July 15, 2026?"
  • "What website hosts the first Google Image result for 'Earthrise Apollo 8 photo'?"
  • "What is the current price of Sony WH-1000XM5 headphones on Google Shopping?"