Benchmark report
Task verified
3m 14s elapsed
The saved attempt met the documented success criterion.
98/ 100
Agent-Ready- AInstall
- SAuth
- SExecute
- SDocs
- SCurrent
Only the upfront docs read and the credential request; the self-serve key path in auth.md is a WorkOS ID-JAG agent flow needing a Firecrawl-trusted client_id.
Where the time went
3:14 agent time- Research0:12
- Attempt1:37
- Verify1:25
Judge reason
Saved code /workspace/home/firecrawl-search/search.py POSTs the exact query "PostgreSQL transaction isolation levels" with {"sources":["web"],"limit":5} to the documented operation POST https://api.firecrawl.dev/v2/search using a Bearer fc- key read at runtime from the benchmark-supplied file (no mocks, no hard-coded data). response_meta.json records HTTP 200 with live server headers (Date Mon, 16 Nov 2026 15:21:29 GMT, X-Response-Time 349.322ms, Etag) and the exact request body; response_body.json is the documented shape {success:true, data:{web:[{url,title,description,position}]}} with 5 results, creditsUsed 2 and request id 01a1e4f2-f0be-70e5-8aa5-98a17fc09f3c. First result has nonempty title "Documentation: 18: 13.2. Transaction Isolation", absolute URL https://www.postgresql.org/docs/current/transaction-iso.html and a 7783-char nonempty excerpt in the documented per-result "description" field; I confirmed title, url and the full excerpt in run_output.txt and first_result.json byte-match data.web[0] of the saved response. Transcript shows one live run (no retries, no scraping substitute, no local search).
Attempt record
- 0:43Answered from saved key: FIRECRAWL_API_KEYno wait
- —First success verified
Attempt artifacts saved.