Benchmark report

Task verified

3m 22s elapsed

The saved attempt met the documented success criterion.

Experience score

100/ 100

Agent-Ready
  • SInstall
  • SAuth
  • SExecute
  • SDocs
  • SCurrent
Experience measures friction in the tested task. The verdict says whether it succeeded.
Agent profile for this run: Install S, Auth S, Execute S, Docs S, Current SSInstallSAuthSExecuteSDocsSCurrent
Agent profile · this run
Judge's note · what slowed the agent most
Essentially nothing: docs fetch, key request, and the single map call completed in about 50 seconds with no retries or errors.

Where the time went

3:22 agent time
  • Starting0:02
  • Research0:16
  • Attempt1:33
  • Verify1:32
3m 22sTotal elapsed
0:50Time to first success
0:00Waited on the owner

Judge reason

Evidence shows one live authenticated call to Firecrawl's documented URL-discovery endpoint POST https://api.firecrawl.dev/v2/map (operationId mapUrls, not a search endpoint). firecrawl-map/map_postgres_docs.py posts {url: https://www.postgresql.org/docs/17/, search: 'tutorial', limit: 30, includeSubdomains: false} with a Bearer fc- key read from a private file, makes exactly one urllib request, and prints len(links) and links[:10] straight from the response with no per-page fetching and no local filtering. evidence/run_output.txt records HTTP 200, 'URL count: 30' and 10 URLs; evidence/map_response.json independently parses to success=true, 30 link objects with real PostgreSQL titles/descriptions, 0 duplicates, and all 30 URLs under https://www.postgresql.org/docs/17/ — the printed first 10 match the file exactly. The saved docs snapshot (evidence/map-endpoint.md, v2 OpenAPI) documents both `search` and `limit` on /map and carries no beta/preview/early-access marking. Transcript shows only /v2/map was called, 200 on the first attempt, no errors or fabrication.

Attempt record

  1. 0:53Answered from saved key: FIRECRAWL_API_KEYno wait
  2. —First success verified

Attempt artifacts saved.