Benchmark report

Task verified

10m 38s elapsed

The saved attempt met the documented success criterion.

Experience score

100/ 100

Agent-Ready
  • SInstall
  • SAuth
  • SExecute
  • SDocs
  • SCurrent
Experience measures friction in the tested task. The verdict says whether it succeeded.
Agent profile for this run: Install S, Auth S, Execute S, Docs S, Current SSInstallSAuthSExecuteSDocsSCurrent
Agent profile · this run
Judge's note · what slowed the agent most
Almost nothing; only wrinkle was the reference not specifying whether schema output is a JSON string or object, which the agent handled defensively in one line.

Where the time went

10:39 agent time
  • Starting0:01
  • Research0:19
  • Attempt5:18
  • Verify5:00
10m 38sTotal elapsed
0:54Time to first success
0:00Waited on the owner

Judge reason

extract.py POSTs {urls:[https://www.postgresql.org/docs/17/transaction-iso.html], summary:{query, schema}} to the documented https://api.exa.ai/contents with x-api-key, no local HTML parsing and no agent-side LLM. Transcript idx 20-21 shows the live run: exit=0, stderr 'HTTP status: 200'. response_full.json is the raw body (requestId b5c30d05..., statuses success/cached, costDollars 0.001, searchTime 1727.6ms) and its results[0].summary decodes to exactly 4 levels whose 16 cells match Table 13.1 verbatim (Read uncommitted: Allowed-but-not-in-PG/Possible/Possible/Possible; Read committed: Not possible/Possible/Possible/Possible; Repeatable read: Not possible/Not possible/Allowed-but-not-in-PG/Possible; Serializable: all Not possible); extracted_object.json and stdout are byte-identical to that decode. Fetched docs (idx 7) show summary.schema is documented and explicitly not beta (only highlights.verbosity/dynamic are beta), so the stop-on-beta clause does not apply.

Attempt record

  1. 1:31Answered from saved key: Exa API keyno wait
  2. —First success verified

Attempt artifacts saved.