Benchmark report

Task verified

6m 16s elapsed

The saved attempt met the documented success criterion.

Experience score

80/ 100

Needs Alignment
  • SInstall
  • SAuth
  • BExecute
  • SDocs
  • SCurrent
Experience measures friction in the tested task. The verdict says whether it succeeded.
Agent profile for this run: Install S, Auth S, Execute B, Docs S, Current SSInstallSAuthBExecuteSDocsSCurrent
Agent profile · this run
Judge's note · what slowed the agent most
Exa documents no URL path/prefix/regex filter for crawls, so the agent had to enumerate the whole OpenAPI spec and run live calls to prove the gap.

Where the time went

6:17 agent time
  • Starting0:01
  • Research0:22
  • Attempt3:48
  • Verify2:06
6m 16sTotal elapsed
—Time to first success
0:00Waited on the owner

Judge reason

Resolved on the task's documented-gap branch, independently supported. The agent enumerated every endpoint in Exa's OpenAPI spec (transcript grep of exa-spec.yaml: /search, /contents, /answer, /findSimilar, /monitors, /agent/runs, /batches, /v0/websets/*, /v0/imports/*) and the saved docs show the only crawl controls are `subpages` (int 0-100, 'The number of subpages to crawl', page cap) and `subpageTarget` ('Term to find specific subpages'; quickstart line 269 of docs-contents-quickstart.md: 'Add subpageTarget when you want Exa to prioritize particular site sections') - a ranking/targeting term, with no path include, prefix or regex filter anywhere; includeDomains/excludeDomains exist only on /search and are domain-level. Live authenticated evidence is genuine, not mocked: evidence/response-baseline.json and response-subpagetarget.json carry HTTP 200, Cloudflare Cf-Ray/X-Request-Id headers, costDollars 0.026 and 25 real subpages each; crawl.py fixes PAGE_CAP = 25 before the request and report.py prints every returned page unfiltered (26 pages, only 1 under /docs/17/tutorial in both calls). No fabrication, no local filtering, no per-page fetching. The second call used subpageTarget once purely to demonstrate it does not restrict URLs (reported as 1/26 in-section, never claimed as success), a minor literal deviation from 'do not run a tuned crawl' that strengthens rather than games the gap report. Noted: result.json references evidence/exa-spec.yaml, which is not present in the saved home, but the transcript's grep output over that 411216-byte download substantiates the endpoint enumeration.

Attempt record

  1. 1:34Answered from saved key: EXA_API_KEYno wait
  2. —First success verified

Attempt artifacts saved.