Benchmark report

Task verified

11m 5s elapsed

The saved attempt met the documented success criterion.

Experience score

100/ 100

Agent-Ready
  • SInstall
  • SAuth
  • SExecute
  • SDocs
  • SCurrent
Experience measures friction in the tested task. The verdict says whether it succeeded.
Agent profile for this run: Install S, Auth S, Execute S, Docs S, Current SSInstallSAuthSExecuteSDocsSCurrent
Agent profile · this run
Judge's note · what slowed the agent most
Almost nothing — the only extra step was a second doc fetch because the POST batch-scrape page does not document the GET status endpoint; whole run took ~74s.

Where the time went

11:05 agent time
  • Starting0:01
  • Research2:06
  • Attempt5:14
  • Verify3:44
11m 5sTotal elapsed
1:14Time to first success
0:00Waited on the owner

Judge reason

Saved code (firecrawl-batch/batch_scrape.py) makes ONE POST https://api.firecrawl.dev/v2/batch/scrape with all five URLs (stdlib urllib, bearer key read from the private benchmark file), then polls GET /v2/batch/scrape/{id} in a loop against TERMINAL={completed,failed,cancelled}. run_output.txt shows HTTP 200, job id 01a1c74c-bf93-72e7-b459-7d943d3fd7f1, polls scraping(0/5) -> scraping(5/5) -> completed(5/5). Independent re-parse of status_response.json confirms status=completed, total=5, completed=5, creditsUsed=5, next=null, and exactly 5 data items whose metadata.sourceURL covers each requested URL with statusCode 200 and markdown lengths 5432/30418/23790/10903/11847 — byte-identical to the printed lengths and to result.json. Markdown is genuine live PostgreSQL 17 doc content (real og: metadata, 'PostgreSQL 19 RC 1 Released' banner), not mocked or hard-coded; no SDK stub or fixture exists. No per-URL single-page calls and no synchronous multi-URL endpoint were used. Agent also checked and documented that batch scrape is not marked beta (only the unrelated onlyCleanContent option is).

Attempt record

  1. 4:24Answered from saved key: Firecrawl API keyno wait
  2. —First success verified

Attempt artifacts saved.