Benchmark report

Task verified

14m 57s elapsed

The saved attempt met the documented success criterion.

Experience score

91/ 100

Agent-Ready
  • SInstall
  • SAuth
  • SExecute
  • BDocs
  • ACurrent
Experience measures friction in the tested task. The verdict says whether it succeeded.
Agent profile for this run: Install S, Auth S, Execute S, Docs B, Current ASInstallSAuthSExecuteBDocsACurrent
Agent profile · this run
Judge's note · what slowed the agent most
subpageTarget only prioritizes rather than restricts, and the cached crawl surfaced few in-section links, forcing repeated live experiments to get an all-in-scope result.

Where the time went

14:57 agent time
  • Starting0:01
  • Research1:10
  • Attempt9:28
  • Verify4:18
14m 57sTotal elapsed
4:15Time to first success
0:00Waited on the owner

Judge reason

Evidence shows one live, authenticated product-side crawl: crawl.py POSTs https://api.exa.ai/contents with urls=[https://www.postgresql.org/docs/17/tutorial.html], subpages=23 (23+start=24 pages, <=25) and the documented include option subpageTarget="/docs/17/tutorial", key read from the benchmark-provided private file via x-api-key. The saved response.json (requestId e93d589b5b6fe9c7d087e2c5480cf787, statuses source=crawled, costDollars 0.024) holds 24 distinct pages, all with URLs under https://www.postgresql.org/docs/17/tutorial; transcript idx 160-161 shows the single run that produced both response.json and crawl-output.txt. I re-derived page list from response.json: printed output matches the saved response one-for-one, and each page's text begins with its title and correct section heading (e.g. tutorial-arch.html -> '1.2. Architectural Fundamentals'), matching the real PostgreSQL 17 tutorial TOC. No per-page fetching, no client link-following, no local filtering of returned pages. Earlier runs with a looser cap/cached crawl returned out-of-scope pages; the agent retuned documented parameters rather than filtering results.

Attempt record

  1. 5:47Answered from saved key: Exa API keyno wait
  2. —First success verified

Attempt artifacts saved.