Benchmark report

Task verified

11m 51s elapsed

The saved attempt met the documented success criterion.

Experience score

95/ 100

Agent-Ready
  • SInstall
  • SAuth
  • SExecute
  • ADocs
  • ACurrent
Experience measures friction in the tested task. The verdict says whether it succeeded.
Agent profile for this run: Install S, Auth S, Execute S, Docs A, Current ASInstallSAuthSExecuteADocsACurrent
Agent profile · this run
Judge's note · what slowed the agent most
Having to recognize that the plan's /extract endpoint is deprecated and beta, then re-fetch the scrape docs to confirm the JSON format carries no beta marking.

Where the time went

11:52 agent time
  • Starting1:05
  • Research1:11
  • Attempt5:17
  • Verify4:20
11m 51sTotal elapsed
1:09Time to first success
0:00Waited on the owner

Judge reason

Saved code firecrawl-extract/extract.py POSTs the fixed URL https://www.postgresql.org/docs/17/transaction-iso.html plus the exact required JSON schema (levels[] with name/dirtyRead/nonrepeatableRead/phantomRead/serializationAnomaly) to Firecrawl's documented structured-extraction operation, POST https://api.firecrawl.dev/v2/scrape with formats:[{type:json,schema,prompt}], authed by Bearer key from a private file; no local HTML parsing and no own LLM. evidence/full_response.json records httpStatus 200, success:true, and Firecrawl-side metadata proving a real live call (scrapeId 01a1c74f-1e1a-7698-a899-12f55834d379, creditsUsed 5, cacheState hit, cachedAt 2026-11-10T20:59:35Z, sourceURL the target page, page statusCode 200). data.json holds exactly four levels matching Table 13.1: Read uncommitted = Allowed-but-not-in-PG/Possible/Possible/Possible; Read committed = Not possible/Possible/Possible/Possible; Repeatable read = Not possible/Not possible/Allowed-but-not-in-PG/Possible; Serializable = Not possible x4. run_stdout.txt shows the same object printed by the run at transcript idx 34-37. The deprecated+beta /v2/extract endpoint was correctly avoided in favor of the unmarked GA /v2/scrape JSON format, so the beta stop-rule did not apply.

Attempt record

  1. 3:53Answered from saved key: FIRECRAWL_API_KEYno wait
  2. —First success verified

Attempt artifacts saved.