Benchmark report

Task verified

3m 14s elapsed

The saved attempt met the documented success criterion.

Experience score

98/ 100

Agent-Ready
  • AInstall
  • SAuth
  • SExecute
  • SDocs
  • SCurrent
Experience measures friction in the tested task. The verdict says whether it succeeded.
Agent profile for this run: Install A, Auth S, Execute S, Docs S, Current SAInstallSAuthSExecuteSDocsSCurrent
Agent profile · this run
Judge's note · what slowed the agent most
Only the upfront docs read and the credential request; the self-serve key path in auth.md is a WorkOS ID-JAG agent flow needing a Firecrawl-trusted client_id.

Where the time went

3:14 agent time
  • Research0:12
  • Attempt1:37
  • Verify1:25
3m 14sTotal elapsed
0:45Time to first success
0:00Waited on the owner

Judge reason

Saved code /workspace/home/firecrawl-search/search.py POSTs the exact query "PostgreSQL transaction isolation levels" with {"sources":["web"],"limit":5} to the documented operation POST https://api.firecrawl.dev/v2/search using a Bearer fc- key read at runtime from the benchmark-supplied file (no mocks, no hard-coded data). response_meta.json records HTTP 200 with live server headers (Date Mon, 16 Nov 2026 15:21:29 GMT, X-Response-Time 349.322ms, Etag) and the exact request body; response_body.json is the documented shape {success:true, data:{web:[{url,title,description,position}]}} with 5 results, creditsUsed 2 and request id 01a1e4f2-f0be-70e5-8aa5-98a17fc09f3c. First result has nonempty title "Documentation: 18: 13.2. Transaction Isolation", absolute URL https://www.postgresql.org/docs/current/transaction-iso.html and a 7783-char nonempty excerpt in the documented per-result "description" field; I confirmed title, url and the full excerpt in run_output.txt and first_result.json byte-match data.web[0] of the saved response. Transcript shows one live run (no retries, no scraping substitute, no local search).

Attempt record

  1. 0:43Answered from saved key: FIRECRAWL_API_KEYno wait
  2. —First success verified

Attempt artifacts saved.