Benchmark report
Task verified
3m 22s elapsed
The saved attempt met the documented success criterion.
100/ 100
Agent-Ready- SInstall
- SAuth
- SExecute
- SDocs
- SCurrent
Essentially nothing: docs fetch, key request, and the single map call completed in about 50 seconds with no retries or errors.
Where the time went
3:22 agent time- Starting0:02
- Research0:16
- Attempt1:33
- Verify1:32
Judge reason
Evidence shows one live authenticated call to Firecrawl's documented URL-discovery endpoint POST https://api.firecrawl.dev/v2/map (operationId mapUrls, not a search endpoint). firecrawl-map/map_postgres_docs.py posts {url: https://www.postgresql.org/docs/17/, search: 'tutorial', limit: 30, includeSubdomains: false} with a Bearer fc- key read from a private file, makes exactly one urllib request, and prints len(links) and links[:10] straight from the response with no per-page fetching and no local filtering. evidence/run_output.txt records HTTP 200, 'URL count: 30' and 10 URLs; evidence/map_response.json independently parses to success=true, 30 link objects with real PostgreSQL titles/descriptions, 0 duplicates, and all 30 URLs under https://www.postgresql.org/docs/17/ — the printed first 10 match the file exactly. The saved docs snapshot (evidence/map-endpoint.md, v2 OpenAPI) documents both `search` and `limit` on /map and carries no beta/preview/early-access marking. Transcript shows only /v2/map was called, 200 on the first attempt, no errors or fabrication.
Attempt record
- 0:53Answered from saved key: FIRECRAWL_API_KEYno wait
- —First success verified
Attempt artifacts saved.