Benchmark report
Task verified
2m 43s elapsed
The saved attempt met the documented success criterion.
100/ 100
Agent-Ready- SInstall
- SAuth
- SExecute
- SDocs
- SCurrent
Essentially nothing: first success at 46s after two doc fetches, one key request, and a single successful live run.
Where the time went
2:44 agent time- Research0:11
- Attempt1:21
- Verify1:11
Judge reason
Saved code /workspace/home/exa-search/search.py POSTs to the documented endpoint https://api.exa.ai/search with x-api-key auth and body query="PostgreSQL transaction isolation levels", contents.highlights (per exa.ai/docs/reference/search). response.json records httpStatus 200 with a live Exa payload (requestId e3049c9c3fd45385188fd1882f8fc811, searchTime 1372.9, costDollars.search.neural 0.007) and 5 distinct results in the documented shape (id/title/url/text/highlights). First result: title "PostgreSQL: Documentation: 18: 13.2. Transaction Isolation", absolute URL https://www.postgresql.org/docs/current/transaction-iso.html, highlights[0] a 773-char nonempty excerpt; run-output.txt printed exactly those three values and the excerpt string matches response.json byte-for-byte. No mocking or hard-coding: the key came from benchmark-request and was read from a private file at runtime.
Attempt record
- 0:42Answered from saved key: EXA_API_KEYno wait
- —First success verified
Attempt artifacts saved.