Benchmark report

Task verified

5m 8s elapsed

The saved attempt met the documented success criterion.

Experience score

98/ 100

Agent-Ready
  • AInstall
  • SAuth
  • SExecute
  • SDocs
  • SCurrent
Experience measures friction in the tested task. The verdict says whether it succeeded.
Agent profile for this run: Install A, Auth S, Execute S, Docs S, Current SAInstallSAuthSExecuteSDocsSCurrent
Agent profile · this run
Judge's note · what slowed the agent most
Crawl results expose only url+raw_content with no title field, so the agent needed two extra runs to parse markdown headings and strip '[#](#ANCHOR)' permalinks.

Where the time went

5:09 agent time
  • Starting0:01
  • Research0:26
  • Attempt2:54
  • Verify1:49
5m 8sTotal elapsed
1:49Time to first success
0:00Waited on the owner

Judge reason

Verified saved evidence, not prose. crawl_pg_tutorial.py calls TavilyClient.crawl once (tavily-python 0.8.5, Authorization: Bearer key from benchmark-request); request.json shows url=https://www.postgresql.org/docs/17/tutorial.html, select_paths=["^/docs/17/tutorial.*"] (the documented include option per the fetched POST /crawl reference), limit=25. response.json is a live 200 body (request_id f727ba2d-..., response_time 0.45s, usage credits 7) with 23 results, 23 distinct URLs, 0 outside https://www.postgresql.org/docs/17/tutorial; raw_content carries live PG site text (Oct 2026 PG19 RC1 banner). crawl_output.txt/pages.json order and URLs match response.json exactly, and every printed title appears verbatim in that page's returned raw_content (e.g. tutorial-window -> '3.5. Window Functions', tutorial-sql -> 'Chapter 2. The SQL Language'), all slug-consistent with the real PG 17 tutorial. No per-page fetching, no local filtering, no mock; endpoint is synchronous so no polling needed and docs show no beta/preview marking.

Attempt record

  1. 1:02Answered from saved key: Tavily API keyno wait
  2. —First success verified

Attempt artifacts saved.