Benchmark report

Task verified

10m 40s elapsed

The saved attempt met the documented success criterion.

Experience score

100/ 100

Agent-Ready
  • SInstall
  • SAuth
  • SExecute
  • SDocs
  • SCurrent
Experience measures friction in the tested task. The verdict says whether it succeeded.
Agent profile for this run: Install S, Auth S, Execute S, Docs S, Current SSInstallSAuthSExecuteSDocsSCurrent
Agent profile · this run
Judge's note · what slowed the agent most
Essentially nothing; the only wrinkle was the enum naming - 'research paper' is not a category value, so the docs had to be read to pick 'publication'.

Where the time went

10:40 agent time
  • Starting0:01
  • Research0:15
  • Attempt5:22
  • Verify5:02
10m 40sTotal elapsed
0:52Time to first success
0:00Waited on the owner

Judge reason

Saved code /workspace/home/exa-papers/search_papers.py POSTs to the documented host https://api.exa.ai/search with x-api-key and body {"query":"serializable snapshot isolation","type":"auto","category":"publication","numResults":10} - no site: operators, domain lists, paper words, or local filtering. The fetched official reference (transcript entry 7) documents the category enum as company|publication|news|personal site|financial report|people and states "research paper" is not in the enum, directing research papers/preprints/journal articles to category=publication, so the required research vertical parameter is present and documented (not beta/gated). response.json records a live HTTP 200 with requestId 68f4e1bd6d1e113bf07305ebf8cd08cb, searchTime 502.1ms, costDollars 0.007 (neural) and 10 results; first three are academic: 'Declarative Serializable Snapshot Isolation' (doi.org/10.1007/978-3-642-23737-9_13, Springer LNCS), 'Serializable snapshot isolation in PostgreSQL | Proc. VLDB' (dl.acm.org/doi/10.14778/2367502.2367523) and 'Precisely Serializable Snapshot Isolation (PSSI) | IEEE ICDE 2011' (dl.acm.org/doi/10.1109/ICDE.2011.5767853). stdout.txt titles/URLs match response.json line for line; nothing mocked or hard-coded - the key came from benchmark-request (outcome "provided") and was read from a private file path, never printed.

Attempt record

  1. 2:59Answered from saved key: Exa API keyno wait
  2. —First success verified

Attempt artifacts saved.