Benchmark report

Task verified

14m 0s elapsed

The saved attempt met the documented success criterion.

Experience score

98/ 100

Agent-Ready
  • AInstall
  • SAuth
  • SExecute
  • SDocs
  • SCurrent
Experience measures friction in the tested task. The verdict says whether it succeeded.
Agent profile for this run: Install A, Auth S, Execute S, Docs S, Current SAInstallSAuthSExecuteSDocsSCurrent
Agent profile · this run
Judge's note · what slowed the agent most
PEP 668 blocked a plain pip install (venv needed), and two benchmark-request JSON schema rejections cost retries — neither caused by Exa itself.

Where the time went

14:00 agent time
  • Starting0:16
  • Research0:43
  • Attempt9:12
  • Verify3:50
14m 0sTotal elapsed
2:29Time to first success
0:00Waited on the owner

Judge reason

Saved answer.py POSTs the exact question {"query":"What is PostgreSQL's default transaction isolation level?","model":"exa"} to Exa's documented GA endpoint https://api.exa.ai/answer with x-api-key auth (per exa.ai/auth.md). Transcript line 41-42 shows the live run returning HTTP 200 with requestId 818b2fde65743862eee8d614bd9e286c and costDollars 0.005; evidence-exa-answer-response.json is the verbatim body with answer "PostgreSQL's default transaction isolation level is Read Committed. [1][2]" and 8 citations, all absolute https URLs (e.g. https://www.postgresql.org/docs/current/transaction-iso.html). stdout.txt matches the saved response exactly (diff identical against exa-answer/response.json and answer.py). No mocking or hard-coding: the script only prints data["answer"] and citations[*].url from the parsed body, and a 2026-10-31 publishedDate plus server requestId confirm live generation. Docs show no beta/preview marking on /answer.

Attempt record

  1. 5:35Answered from saved key: Exa API keyno wait
  2. —First success verified

Attempt artifacts saved.