Benchmark report

Task verified

13m 38s elapsed

The saved attempt met the documented success criterion.

Experience score

98/ 100

Agent-Ready
  • AInstall
  • SAuth
  • SExecute
  • SDocs
  • SCurrent
Experience measures friction in the tested task. The verdict says whether it succeeded.
Agent profile for this run: Install A, Auth S, Execute S, Docs S, Current SAInstallSAuthSExecuteSDocsSCurrent
Agent profile · this run
Judge's note · what slowed the agent most
Only real friction was the PEP 668 externally-managed Python blocking `pip install firecrawl-py`, forcing a venv; environment issue, resolved in one command.

Where the time went

13:38 agent time
  • Starting0:07
  • Research1:21
  • Attempt7:38
  • Verify4:32
13m 38sTotal elapsed
1:28Time to first success
0:00Waited on the owner

Judge reason

Saved code /workspace/home/firecrawl-map-task/map_postgres_docs.py makes exactly one documented discovery call via firecrawl-py 4.55.0 (Firecrawl(api_key=...).map(url='https://www.postgresql.org/docs/17/', search='tutorial', limit=100)) wrapping POST https://api.firecrawl.dev/v2/map, with the key read from a private file at runtime - no mock, no hard-coded data, no per-page fetch or scrape call, and no local filtering of links. The saved full response evidence_map_response.json (identical to firecrawl-map-task/map_response.json) contains 100 link objects with real titles/descriptions; independent re-check confirms count=100, 0 URLs outside the https://www.postgresql.org/docs/17/ prefix, and that the first 10 URLs in evidence_stdout.txt (tutorial-views, tutorial-update, tutorial-sql-intro, ...) exactly match the response order. Transcript shows docs fetched first (docs.firecrawl.dev/features/map explicitly not marked beta/preview/request-only; search is a documented parameter so it was set to 'tutorial'), key obtained via benchmark-request (outcome 'provided'), venv install of firecrawl-py, then the live run at idx 41-47 producing the output. benchmark-success (idx 51) was run only after that successful live call and after the evidence file was copied, so the recorded first success is backed by real output.

Attempt record

  1. 6:37Answered from saved key: Firecrawl API keyno wait
  2. —First success verified

Attempt artifacts saved.