Benchmark report
Task verified
12m 45s elapsed
The saved attempt met the documented success criterion.
96/ 100
Agent-Ready- AInstall
- SAuth
- SExecute
- SDocs
- ACurrent
Exa has no map endpoint, and the plan's /contents subpages+subpageTarget route crawled site-wide nav off-prefix, costing ~a third of the turns before the pivot to /search includeDomains.
Where the time went
12:46 agent time- Starting0:01
- Research0:55
- Attempt6:42
- Verify5:07
Judge reason
Verified from saved artifacts, not prose. exa-map/exa_map.py makes ONE live authenticated POST https://api.exa.ai/search (x-api-key from a private key file) with body {"query":"tutorial","includeDomains":["www.postgresql.org/docs/17"],"numResults":50} (request.json). response.json is a genuine Exa response (requestId 9fde00d8d574d8624d3c260fef536aee, searchTime, costDollars {total 0.025, neural}) with 28 results; I recomputed: result.json count/first10/urls == response results[].url exactly, and all 28 URLs start with https://www.postgresql.org/docs/17/. Result objects carry only id/title/url/publishedDate - no text/highlights/contents were requested, so no page fetching or crawling occurs in the code, and the printed list is results[].url verbatim with no local filtering (all_under_prefix true, 0 off-prefix). The 'tutorial' filter is the documented server-side search term, and path scoping via includeDomains is documented verbatim in the saved docs/search.md ('a hostname with a path prefix (for example, example.com/docs) ... results will only come from the matching domains or paths'). Exa publishes no /map endpoint; the agent live-tested the plan's /contents subpages+subpageTarget route (experiments/resp-sub10.json, resp-sub100.json: 100 subpages, only 64 under the prefix, documented as 'the number of subpages to crawl') and documented in NOTES.md why it violates the no-crawl and no-local-filter constraints before using the compliant documented search path. Beta/preview markings were checked and apply only to Dynamic Highlights, not to the parameters used.
Attempt record
- 1:46Answered from saved key: Exa API keyno wait
- —First success verified
Attempt artifacts saved.