Benchmark report

Task verified

12m 45s elapsed

The saved attempt met the documented success criterion.

Experience score

96/ 100

Agent-Ready
  • AInstall
  • SAuth
  • SExecute
  • SDocs
  • ACurrent
Experience measures friction in the tested task. The verdict says whether it succeeded.
Agent profile for this run: Install A, Auth S, Execute S, Docs S, Current AAInstallSAuthSExecuteSDocsACurrent
Agent profile · this run
Judge's note · what slowed the agent most
Exa has no map endpoint, and the plan's /contents subpages+subpageTarget route crawled site-wide nav off-prefix, costing ~a third of the turns before the pivot to /search includeDomains.

Where the time went

12:46 agent time
  • Starting0:01
  • Research0:55
  • Attempt6:42
  • Verify5:07
12m 45sTotal elapsed
3:40Time to first success
0:00Waited on the owner

Judge reason

Verified from saved artifacts, not prose. exa-map/exa_map.py makes ONE live authenticated POST https://api.exa.ai/search (x-api-key from a private key file) with body {"query":"tutorial","includeDomains":["www.postgresql.org/docs/17"],"numResults":50} (request.json). response.json is a genuine Exa response (requestId 9fde00d8d574d8624d3c260fef536aee, searchTime, costDollars {total 0.025, neural}) with 28 results; I recomputed: result.json count/first10/urls == response results[].url exactly, and all 28 URLs start with https://www.postgresql.org/docs/17/. Result objects carry only id/title/url/publishedDate - no text/highlights/contents were requested, so no page fetching or crawling occurs in the code, and the printed list is results[].url verbatim with no local filtering (all_under_prefix true, 0 off-prefix). The 'tutorial' filter is the documented server-side search term, and path scoping via includeDomains is documented verbatim in the saved docs/search.md ('a hostname with a path prefix (for example, example.com/docs) ... results will only come from the matching domains or paths'). Exa publishes no /map endpoint; the agent live-tested the plan's /contents subpages+subpageTarget route (experiments/resp-sub10.json, resp-sub100.json: 100 subpages, only 64 under the prefix, documented as 'the number of subpages to crawl') and documented in NOTES.md why it violates the no-crawl and no-local-filter constraints before using the compliant documented search path. Beta/preview markings were checked and apply only to Dynamic Highlights, not to the parameters used.

Attempt record

  1. 1:46Answered from saved key: Exa API keyno wait
  2. —First success verified

Attempt artifacts saved.