Benchmark report
Task verified
13m 38s elapsed
The saved attempt met the documented success criterion.
98/ 100
Agent-Ready- AInstall
- SAuth
- SExecute
- SDocs
- SCurrent
Only real friction was the PEP 668 externally-managed Python blocking `pip install firecrawl-py`, forcing a venv; environment issue, resolved in one command.
Where the time went
13:38 agent time- Starting0:07
- Research1:21
- Attempt7:38
- Verify4:32
Judge reason
Saved code /workspace/home/firecrawl-map-task/map_postgres_docs.py makes exactly one documented discovery call via firecrawl-py 4.55.0 (Firecrawl(api_key=...).map(url='https://www.postgresql.org/docs/17/', search='tutorial', limit=100)) wrapping POST https://api.firecrawl.dev/v2/map, with the key read from a private file at runtime - no mock, no hard-coded data, no per-page fetch or scrape call, and no local filtering of links. The saved full response evidence_map_response.json (identical to firecrawl-map-task/map_response.json) contains 100 link objects with real titles/descriptions; independent re-check confirms count=100, 0 URLs outside the https://www.postgresql.org/docs/17/ prefix, and that the first 10 URLs in evidence_stdout.txt (tutorial-views, tutorial-update, tutorial-sql-intro, ...) exactly match the response order. Transcript shows docs fetched first (docs.firecrawl.dev/features/map explicitly not marked beta/preview/request-only; search is a documented parameter so it was set to 'tutorial'), key obtained via benchmark-request (outcome 'provided'), venv install of firecrawl-py, then the live run at idx 41-47 producing the output. benchmark-success (idx 51) was run only after that successful live call and after the evidence file was copied, so the recorded first success is backed by real output.
Attempt record
- 6:37Answered from saved key: Firecrawl API keyno wait
- —First success verified
Attempt artifacts saved.