Benchmark report

Task verified

4m 2s elapsed

The saved attempt met the documented success criterion.

Experience score

100/ 100

Agent-Ready
  • SInstall
  • SAuth
  • SExecute
  • SDocs
  • SCurrent
Experience measures friction in the tested task. The verdict says whether it succeeded.
Agent profile for this run: Install S, Auth S, Execute S, Docs S, Current SSInstallSAuthSExecuteSDocsSCurrent
Agent profile · this run
Judge's note · what slowed the agent most
Essentially nothing: only the upfront doc fetches and a one-shot SDK install; the first live request already satisfied every assertion.

Where the time went

4:03 agent time
  • Starting0:01
  • Research0:12
  • Attempt2:15
  • Verify1:36
4m 2sTotal elapsed
0:57Time to first success
0:00Waited on the owner

Judge reason

Transcript shows a genuine live run, not prose. search_firecrawl.py (full source saved in transcript idx 29) is stdlib urllib doing POST https://api.firecrawl.dev/v2/search with payload {"query": "PostgreSQL transaction isolation levels", "limit": 5} and Authorization: Bearer from a key file (benchmark-request outcome 'provided'); no mock or hard-coded data. Run output (idx 34, EXIT=0): 'HTTP status: 200', 'success: True web results: 5', first result title 'Documentation: 18: 13.2. Transaction Isolation', url https://www.postgresql.org/docs/current/transaction-iso.html, a multi-thousand-char excerpt, then 'assertions passed: nonempty title, absolute http(s) url, nonempty excerpt'. Re-reading the saved raw body with jq (idx 45) independently confirms success:true, data.web length 5, position 1, description length 7783 and the same title/url, matching the printed values and the documented shape {success, data.web[]{url,title,description,position}}. The excerpt field is documented: fetched fc_highlights.md states highlights are enabled by default on /v2/search and returned in 'description' for web results, so no API gap applies. The official firecrawl Python SDK installed into ./vendor (EXIT=0) and firecrawl.search(query='PostgreSQL transaction isolation levels', limit=5) reproduced the identical first result (idx 54, EXIT=0). The project dir in the saved home snapshot shows only a partially captured vendor/ tree (snapshot truncated at 100 files), but the saved commands and their outputs establish the result.

Attempt record

  1. 0:42Answered from saved key: FIRECRAWL_API_KEYno wait
  2. —First success verified

Attempt artifacts saved.