Benchmark report

Task verified

12m 22s elapsed

The saved attempt met the documented success criterion.

Experience score

94/ 100

Agent-Ready
  • AInstall
  • SAuth
  • SExecute
  • ADocs
  • SCurrent
Experience measures friction in the tested task. The verdict says whether it succeeded.
Agent profile for this run: Install A, Auth S, Execute S, Docs A, Current SAInstallSAuthSExecuteADocsSCurrent
Agent profile · this run
Judge's note · what slowed the agent most
Fetching the agent docs as HTML first returned 519KB of page markup, costing an extra round of .md/llms.txt fetches before the right spec was found.

Where the time went

12:22 agent time
  • Starting0:14
  • Research3:08
  • Attempt6:43
  • Verify2:18
12m 22sTotal elapsed
2:41Time to first success
0:00Waited on the owner

Judge reason

Saved code firecrawl-cited-answer/ask_agent.py makes a real HTTP call (urllib, Bearer key read from a private file) to Firecrawl's documented answer operation POST https://api.firecrawl.dev/v2/agent, then polls GET /v2/agent/{jobId}. run.log shows the live sequence: POST -> 200 with job id 01a1c755-0430-75ab-973f-9a0c9f6c58cb, 11 polls status=processing, then status=completed. The saved final_response.json is the product's response: data.answer = "PostgreSQL's default transaction isolation level is Read Committed (READ COMMITTED)." with data.citations[0].url = https://www.postgresql.org/docs/current/transaction-iso.html (absolute https), model spark-2, mode extract. The printed ANSWER/CITATIONS block in run.log matches the saved JSON byte-for-byte; nothing is mocked or hard-coded, and the answer text comes from Firecrawl, not the attempt agent. The exact campaign question was sent in the request prompt (request.json). Docs snapshots (features/agent.md, api-reference/endpoint/agent.md, v2-openapi.json) show /agent documented with bearerAuth and no beta/preview/early-access/request-access marking, so the stop condition did not apply; the research plan itself designated /v2/agent as the citation-bearing answer capability. Key was obtained via benchmark-request (outcome provided) and never printed.

Attempt record

  1. 5:59Answered from saved key: FIRECRAWL_API_KEYno wait
  2. —First success verified

Attempt artifacts saved.