The Prompting Company

Benchmark report

Task verified

2m 50s elapsed

The saved attempt met the documented success criterion.

Experience score

94/ 100

Agent-Ready
  • AInstall
  • SAuth
  • SExecute
  • ADocs
  • SCurrent
Experience measures friction in the tested task. The verdict says whether it succeeded.
Agent profile for this run: Install A, Auth S, Execute S, Docs A, Current SAInstallSAuthSExecuteADocsSCurrent
Agent profile · this run
Judge's note · what slowed the agent most
The PEP 668 pip block and a 734KB HTML docs page dump before switching to docs.exa.ai/reference/search.md cost the only detours; total time to success was ~65s.

Where the time went

2:50 agent time
  • Starting0:01
  • Research0:11
  • Attempt1:50
  • Verify0:48
2m 50sTotal elapsed
1:05Time to first success
0:00Waited on the owner

Judge reason

Saved code exa-search/search.py sends the exact query "PostgreSQL transaction isolation levels" via official exa-py 2.25.0 (exa.search) and search_http.py posts it to the documented endpoint https://api.exa.ai/search with x-api-key. Transcript shows pip PEP668 failure, then venv install reporting 'exa-py 2.25.0', a real key obtained via benchmark-request (file path returned, never hard-coded), and live runs: SDK stdout 'First result title: PostgreSQL: Documentation: 18: 13.2. Transaction Isolation / URL https://www.postgresql.org/docs/current/transaction-iso.html' (exit 0), and raw REST run printing 'HTTP status: 200, result count: 5'. Saved http_response.json is a genuine Exa payload (requestId ca019c7f7b3ffa3dfc24ee9a731615e8, searchTime 359.4, costDollars 0.007 neural, 5 results) in the documented shape, and the printed first title/URL match results[0] in both sdk_response.json and http_response.json. No mocking or hard-coded response. benchmark-success was recorded at 65s referencing evidence-first-success.txt, written after the successful SDK run whose nonempty live results were already in context, so the planned operation had genuinely succeeded when marked.

Attempt record

  1. 0:58Answered from saved key: Exa API keyno wait
  2. —First success verified

Attempt artifacts saved.