Benchmark report
Task verified
4m 15s elapsed
The saved attempt met the documented success criterion.
100/ 100
Agent-Ready- SInstall
- SAuth
- SExecute
- SDocs
- SCurrent
Almost nothing; only the job briefly reporting status=scraping at completed=5/5, costing one extra 5s poll before it flipped to completed.
Where the time went
4:16 agent time- Research0:14
- Attempt2:16
- Verify1:45
Judge reason
Independently verified live async batch job. Saved code (firecrawl-batch/batch_scrape.py) POSTs all five URLs in ONE request to the documented async endpoint https://api.firecrawl.dev/v2/batch/scrape; 01_submit_response.json shows HTTP 200 with a single job id 01a1e49c-d48f-758c-b8d6-435a89a5136f, invalidURLs=[], and a real server Date header (Mon, 16 Nov 2026 13:47:25 GMT). The code then polls GET /v2/batch/scrape/{id} in a loop against TERMINAL={completed,failed,cancelled}; 02_poll_log.json records three real polls (scraping 0/5 -> scraping 5/5 -> completed 5/5). Recomputing from the raw 03_final_job_response.json myself: status=completed, total=5, completed=5, next=null, creditsUsed=5, and exactly 5 data items whose sourceURLs are the five requested URLs, each statusCode 200 with markdown lengths 5499/30485/23857/10970/11914 - matching the printed run_output.txt exactly. Content is genuine and distinct (5 unique SHA-256 hashes) and topically correct per URL (explicit-locking has 285 'lock' hits, tutorial-window 89 'window' hits, mvcc-intro 34 'mvcc' hits), so it is not mocked or hard-coded. No looped single-page calls and no synchronous multi-URL call were used. Timestamps are coherent with real async behavior: createdAt 13:47:25.744, completedAt 13:47:28.584, polls at 13:47:25.9/31.3/36.6. Transcript row 27 shows the single live run with EXIT=0 and output byte-identical to the saved file. Docs fetch explicitly reported no beta/preview marking, so the capability is GA.
Attempt record
- 1:00Answered from saved key: Firecrawl API keyno wait
- —First success verified
Attempt artifacts saved.