Benchmark report
Task verified
14m 0s elapsed
The saved attempt met the documented success criterion.
98/ 100
Agent-Ready- AInstall
- SAuth
- SExecute
- SDocs
- SCurrent
PEP 668 blocked a plain pip install (venv needed), and two benchmark-request JSON schema rejections cost retries — neither caused by Exa itself.
Where the time went
14:00 agent time- Starting0:16
- Research0:43
- Attempt9:12
- Verify3:50
Judge reason
Saved answer.py POSTs the exact question {"query":"What is PostgreSQL's default transaction isolation level?","model":"exa"} to Exa's documented GA endpoint https://api.exa.ai/answer with x-api-key auth (per exa.ai/auth.md). Transcript line 41-42 shows the live run returning HTTP 200 with requestId 818b2fde65743862eee8d614bd9e286c and costDollars 0.005; evidence-exa-answer-response.json is the verbatim body with answer "PostgreSQL's default transaction isolation level is Read Committed. [1][2]" and 8 citations, all absolute https URLs (e.g. https://www.postgresql.org/docs/current/transaction-iso.html). stdout.txt matches the saved response exactly (diff identical against exa-answer/response.json and answer.py). No mocking or hard-coding: the script only prints data["answer"] and citations[*].url from the parsed body, and a 2026-10-31 publishedDate plus server requestId confirm live generation. Docs show no beta/preview marking on /answer.
Attempt record
- 5:35Answered from saved key: Exa API keyno wait
- —First success verified
Attempt artifacts saved.