Benchmark report

Task verified

4m 0s elapsed

The saved attempt met the documented success criterion.

Experience score

96/ 100

Agent-Ready
  • AInstall
  • SAuth
  • SExecute
  • SDocs
  • ACurrent
Experience measures friction in the tested task. The verdict says whether it succeeded.
Agent profile for this run: Install A, Auth S, Execute S, Docs S, Current AAInstallSAuthSExecuteSDocsACurrent
Agent profile · this run
Judge's note · what slowed the agent most
The PEP 668 externally-managed-environment block on system pip forced an extra venv creation step before the SDK could be installed.

Where the time went

4:00 agent time
  • Research0:11
  • Attempt2:28
  • Verify1:21
4m 0sTotal elapsed
1:24Time to first success
0:00Waited on the owner

Judge reason

Saved code exa-search/search_exa.py POSTs https://api.exa.ai/search with x-api-key auth and body query="PostgreSQL transaction isolation levels", contents.highlights — the documented web search operation. response_meta.json records HTTP 200 with genuine Cloudflare/Exa headers (X-Request-Id 04eb3d5178f00e5561a0f43bd81de982, X-Exa-Region us-west-2, Date Mon, 16 Nov 2026 14:50:25 GMT); response.json has the documented shape (requestId, resolvedSearchType, results[5], searchTime 1602.9, costDollars 0.007) with matching requestId. results[0] has title "PostgreSQL: Documentation: 18: 13.2. Transaction Isolation", url https://www.postgresql.org/docs/current/transaction-iso.html (absolute https), and highlights[0] of 3309 chars. run_output.txt prints exactly those three values and "assertions (title/absolute url/nonempty excerpt): PASS"; the exa-py SDK run reproduced the same first result. Transcript step 36 shows the live run executed in-sandbox with the key read from the benchmark-provided private file — no mock, scrape, or hard-coded data.

Attempt record

  1. 0:47Answered from saved key: EXA_API_KEYno wait
  2. —First success verified

Attempt artifacts saved.