Benchmark report

Task verified

13m 35s elapsed

The saved attempt met the documented success criterion.

Experience score

98/ 100

Agent-Ready
  • AInstall
  • SAuth
  • SExecute
  • SDocs
  • SCurrent
Experience measures friction in the tested task. The verdict says whether it succeeded.
Agent profile for this run: Install A, Auth S, Execute S, Docs S, Current SAInstallSAuthSExecuteSDocsSCurrent
Agent profile · this run
Judge's note · what slowed the agent most
Only minor doc fragmentation: the POST /v2/crawl page lacked the polling endpoint and an auth example, costing one extra doc fetch; the run itself finished in ~84s.

Where the time went

13:36 agent time
  • Starting1:04
  • Research2:13
  • Attempt7:31
  • Verify2:47
13m 35sTotal elapsed
1:24Time to first success
0:00Waited on the owner

Judge reason

Saved request.json/crawl.py show one product-side Firecrawl v2 job: POST https://api.firecrawl.dev/v2/crawl with url=https://www.postgresql.org/docs/17/tutorial.html, documented includePaths ["^/?docs/17/tutorial.*"], limit 25, then polling GET /v2/crawl/{id} (run_output.txt: start id 01a1c74e-15e3-75ff-96ab-c19582fabfa4, polls scraping 0/1 -> 22/24 -> completed 24/24). crawl_response.json is the full live response: status=completed, completed/total 24, creditsUsed 24, 24 data entries, 24 distinct sourceURLs all under https://www.postgresql.org/docs/17/tutorial, each metadata.statusCode 200 with real markdown bodies. I re-derived the list and confirmed result.json and the printed run_output listing match the response exactly, and each title's section text (e.g. '3.5. Window Functions' for tutorial-window.html) appears in that page's own markdown, 24/24. No local link-following, per-page fetching or post-filtering in the code; no mocked or hard-coded data.

Attempt record

  1. 5:47Answered from saved key: Firecrawl API keyno wait
  2. —First success verified

Attempt artifacts saved.