The Prompting Company

Benchmark report

Task verified

8m 35s elapsed

The saved attempt met the documented success criterion.

Experience score

93/ 100

Agent-Ready
  • SInstall
  • AAuth
  • SExecute
  • SDocs
  • SCurrent
Experience measures friction in the tested task. The verdict says whether it succeeded.
Agent profile for this run: Install S, Auth A, Execute S, Docs S, Current SSInstallAAuthSExecuteSDocsSCurrent
Agent profile · this run
Judge's note · what slowed the agent most
The API key handoff: the first benchmark-request timed out with no answer, so most of the ~6.4 min elapsed was waiting on the credential, not on Exa itself.

Where the time went

5:00 on the owner · 3:36 agent time
  • Starting0:01
  • Research0:12
  • Attempt2:22
  • Verify1:01
8m 35sTotal elapsed
6:24Time to first success
5:00Waited on the owner

Judge reason

Live Exa search with highlights verified end to end. Transcript shows pip install exa-py (prints 'exa-py 2.25.0'), then run_search.py calling exa.search('best blog posts about vector databases', contents={'highlights': True}) through the real exa_py package; its stdout (evidence/sdk_output.txt) lists 10 results each with a non-empty title, an https URL (e.g. 'We Tried and Tested 10 Best Vector Databases for RAG ...' / https://www.zenml.io/blog/vector-databases-for-rag) and highlight text. A confirming REST call (POST https://api.exa.ai/search with Bearer auth) saved evidence/curl_raw_response.json ending in HTTP_STATUS:200, with requestId e2e3b51f57f33ad13dd28d1127faca9b, searchTime 7.3 and costDollars 0.007 — server-side fields no mock produced. No stubs or fixtures appear anywhere; the preflight unauthenticated call correctly returned HTTP 402 and keyless mcp.exa.ai returned 403, confirming the live service was being hit.

Attempt record

  1. 6:34Answered from saved key: Exa API keyno wait
  2. —First success verified

Attempt artifacts saved.