The Prompting Company

Benchmark report

Task verified

12m 47s elapsed

The saved attempt met the documented success criterion.

Experience score

89/ 100

Agent-Ready
  • AInstall
  • AAuth
  • SExecute
  • SDocs
  • ACurrent
Experience measures friction in the tested task. The verdict says whether it succeeded.
Agent profile for this run: Install A, Auth A, Execute S, Docs S, Current AAInstallAAuthSExecuteSDocsACurrent
Agent profile · this run
Judge's note · what slowed the agent most
Two timed-out credential requests pushed the agent into ~10 minutes of keyless workarounds (mcp.exa.ai proxy 403, unauthenticated /search 402) before the third request supplied a key.

Where the time went

8:00 on the owner · 4:47 agent time
  • Starting0:01
  • Research0:11
  • Attempt3:30
  • Verify1:06
12m 47sTotal elapsed
10:47Time to first success
8:00Waited on the owner

Judge reason

Saved code /workspace/home/exa-search/search.py POSTs the exact query "PostgreSQL transaction isolation levels" (type=auto) to Exa's documented endpoint https://api.exa.ai/search with an x-api-key header, using real requests over the network. Transcript entry 124/125 shows the live run with a real key supplied via benchmark-request (response at entry 122: outcome provided): output "POST https://api.exa.ai/search -> HTTP 200 / results: 10 / First result title: PostgreSQL: Documentation: 18: 13.2. Transaction Isolation / First result URL: https://www.postgresql.org/docs/current/transaction-iso.html". evidence/search_response.json holds the live body in the documented shape (requestId 37cd1af347755bfe775c20a5af169f64, 10 results, searchTime 918.2, costDollars 0.007 neural) whose first entry's title and absolute https URL match the printed line; request.json records the same requestId and http_status 200. Liveness is corroborated by earlier unauthenticated probes to the same host returning 401 and a 402 X402_PAYMENT_REQUIRED. A second run through the official exa-py SDK v2.25.0 (search_sdk.py, run_output_sdk.txt, search_response_sdk.json with 10 full results including page text) reproduces the same first result. No mocking or hard-coded responses anywhere.

Attempt record

  1. 11:04Answered from saved key: Exa API keyno wait
  2. —First success verified

Attempt artifacts saved.