Benchmark report

Task verified

4m 15s elapsed

The saved attempt met the documented success criterion.

Experience score

100/ 100

Agent-Ready
  • SInstall
  • SAuth
  • SExecute
  • SDocs
  • SCurrent
Experience measures friction in the tested task. The verdict says whether it succeeded.
Agent profile for this run: Install S, Auth S, Execute S, Docs S, Current SSInstallSAuthSExecuteSDocsSCurrent
Agent profile · this run
Judge's note · what slowed the agent most
Almost nothing; only the job briefly reporting status=scraping at completed=5/5, costing one extra 5s poll before it flipped to completed.

Where the time went

4:16 agent time
  • Research0:14
  • Attempt2:16
  • Verify1:45
4m 15sTotal elapsed
1:28Time to first success
0:00Waited on the owner

Judge reason

Independently verified live async batch job. Saved code (firecrawl-batch/batch_scrape.py) POSTs all five URLs in ONE request to the documented async endpoint https://api.firecrawl.dev/v2/batch/scrape; 01_submit_response.json shows HTTP 200 with a single job id 01a1e49c-d48f-758c-b8d6-435a89a5136f, invalidURLs=[], and a real server Date header (Mon, 16 Nov 2026 13:47:25 GMT). The code then polls GET /v2/batch/scrape/{id} in a loop against TERMINAL={completed,failed,cancelled}; 02_poll_log.json records three real polls (scraping 0/5 -> scraping 5/5 -> completed 5/5). Recomputing from the raw 03_final_job_response.json myself: status=completed, total=5, completed=5, next=null, creditsUsed=5, and exactly 5 data items whose sourceURLs are the five requested URLs, each statusCode 200 with markdown lengths 5499/30485/23857/10970/11914 - matching the printed run_output.txt exactly. Content is genuine and distinct (5 unique SHA-256 hashes) and topically correct per URL (explicit-locking has 285 'lock' hits, tutorial-window 89 'window' hits, mvcc-intro 34 'mvcc' hits), so it is not mocked or hard-coded. No looped single-page calls and no synchronous multi-URL call were used. Timestamps are coherent with real async behavior: createdAt 13:47:25.744, completedAt 13:47:28.584, polls at 13:47:25.9/31.3/36.6. Transcript row 27 shows the single live run with EXIT=0 and output byte-identical to the saved file. Docs fetch explicitly reported no beta/preview marking, so the capability is GA.

Attempt record

  1. 1:00Answered from saved key: Firecrawl API keyno wait
  2. —First success verified

Attempt artifacts saved.