Benchmark report

Task verified

5m 13s elapsed

The saved attempt met the documented success criterion.

Experience score

94/ 100

Agent-Ready
  • AInstall
  • SAuth
  • SExecute
  • ADocs
  • SCurrent
Experience measures friction in the tested task. The verdict says whether it succeeded.
Agent profile for this run: Install A, Auth S, Execute S, Docs A, Current SAInstallSAuthSExecuteADocsSCurrent
Agent profile · this run
Judge's note · what slowed the agent most
Sandbox egress blocked direct curl to www.postgresql.org (HTTP 000), so title cross-checks took several failed attempts before falling back to the saved markdown.

Where the time went

5:14 agent time
  • Research0:15
  • Attempt3:10
  • Verify1:48
5m 13sTotal elapsed
1:49Time to first success
0:00Waited on the owner

Judge reason

Saved code (code/crawl_pg_tutorial.py) issues one POST https://api.firecrawl.dev/v2/crawl with url=https://www.postgresql.org/docs/17/tutorial.html, documented includePaths=["^/?docs/17/tutorial"] and limit=25, then only polls GET /v2/crawl/{id}; no local link-following or filtering. evidence/crawl_start_response.json shows HTTP 200 with job id 01a1e520-4006-7056-a54f-6a8a647a7ff5 and live response headers (Date Mon, 16 Nov 2026 16:10:58, X-Response-Time 544ms). evidence/crawl_status_response.json records 5 polls ending status=completed, completed=24/total=24, with 24 data entries. I recomputed from the saved response: 24 distinct sourceURLs, 0 outside the https://www.postgresql.org/docs/17/tutorial prefix, no empty titles, and pages.json/run_output.txt rows are byte-identical to final.data[].metadata.{sourceURL,title}. Titles match page content (each scraped markdown's first heading equals the title's section, e.g. tutorial-agg '2.7. Aggregate Functions'), and I independently fetched https://www.postgresql.org/docs/17/tutorial-window.html (HTTP 200) whose <title> is exactly 'PostgreSQL: Documentation: 17: 3.5. Window Functions', matching the saved record. includePaths and limit are confirmed documented options in the saved crawl-post.md spec. 24 pages is >=10 and <=25.

Attempt record

  1. 1:01Answered from saved key: Firecrawl API keyno wait
  2. —First success verified

Attempt artifacts saved.