Benchmark report

Task verified

10m 23s elapsed

The saved attempt met the documented success criterion.

Experience score

96/ 100

Agent-Ready
  • SInstall
  • SAuth
  • SExecute
  • ADocs
  • SCurrent
Experience measures friction in the tested task. The verdict says whether it succeeded.
Agent profile for this run: Install S, Auth S, Execute S, Docs A, Current SSInstallSAuthSExecuteADocsSCurrent
Agent profile · this run
Judge's note · what slowed the agent most
Disproving a search-summary claim that structured output was beta-gated by fetching and grepping the real doc pages before proceeding.

Where the time went

10:24 agent time
  • Starting0:49
  • Research1:31
  • Attempt4:19
  • Verify3:44
10m 23sTotal elapsed
1:58Time to first success
0:00Waited on the owner

Judge reason

Saved extract.py posts the fixed URL https://www.postgresql.org/docs/17/transaction-iso.html plus the required levels/name/dirtyRead/nonrepeatableRead/phantomRead/serializationAnomaly JSON schema as output_schema to TinyFish's documented endpoint POST https://agent.tinyfish.ai/v1/automation/run with the documented X-API-Key header, using only stdlib urllib (no local scraping, parsing, or own LLM; result read straight from data['result'] with no fallback). evidence/full_response.json records http_status 200, run_id 9a970b6d-76ca-4cf5-9f3f-09869d5460db, status COMPLETED, started_at 2026-11-10T21:35:39.681Z / finished_at 21:35:59.835Z, num_of_steps 3, error null, and exactly four levels whose 16 cell values match Table 13.1 (Read uncommitted: Allowed, but not in PG/Possible/Possible/Possible; Read committed: Not possible/Possible/Possible/Possible; Repeatable read: Not possible/Not possible/Allowed, but not in PG/Possible; Serializable: Not possible x4). The transcript shows the same stdout and 'HTTP 200' from the single live run (key read from the private benchmark-request file, never echoed), and verify.py's saved output independently asserts all four rows. Fetched official doc pages in docs_raw/ contain no beta/preview/early-access/request-access marking for output_schema, so the stop-and-report clause did not apply. The goal prompt supplied only example cell wordings, not the per-row mapping, so the row-to-value assignment was produced by the product.

Attempt record

  1. 3:39Answered from saved key: TinyFish API keyno wait
  2. —First success verified

Attempt artifacts saved.