Benchmark report

Task verified

12m 53s elapsed

The saved attempt met the documented success criterion.

Experience score

90/ 100

Agent-Ready
  • AInstall
  • AAuth
  • SExecute
  • SDocs
  • SCurrent
Experience measures friction in the tested task. The verdict says whether it succeeded.
Agent profile for this run: Install A, Auth A, Execute S, Docs S, Current SAInstallAAuthSExecuteSDocsSCurrent
Agent profile · this run
Judge's note · what slowed the agent most
Only minor: the documented auth ceremony needed a browser claim, and the independent title check via direct postgresql.org fetch was blocked by the proxy.

Where the time went

12:53 agent time
  • Starting0:04
  • Research0:45
  • Attempt7:47
  • Verify4:18
12m 53sTotal elapsed
1:11Time to first success
0:00Waited on the owner

Judge reason

Saved code crawl_tutorial.py posts one synchronous job to the documented POST https://api.context.dev/v1/web/crawl with url=https://www.postgresql.org/docs/17/tutorial.html, maxPages=25 and the documented include filter urlRegex=^https://www\.postgresql\.org/docs/17/tutorial (confirmed as the include option in the saved OpenAPI doc docs/crawl.md lines 76-95, with no beta/preview marking). evidence/run_output.txt shows HTTP 200, page count 24, and server-side scoping metadata {numUrls:24, numSucceeded:24, numFailed:0, numSkipped:659}. evidence/crawl_response_full.json (205 KB) is a genuine live response: request_id 575d6762-ae61-4ef2-908b-8c55fae24473, cache miss, credits_consumed 24, plus per-page Markdown. I re-derived from it 24 distinct URLs, all under https://www.postgresql.org/docs/17/tutorial, all statusCode 200, crawlDepth 0-1; the printed URL/title list matches crawl_pages.json and the response exactly, and each title (e.g. '2.6. Joins Between Tables', '3.5. Window Functions', 'Chapter 3. Advanced Features') appears in that page's own returned Markdown, matching the real PostgreSQL 17 tutorial pages. No per-page fetching, link following, or local filtering: the script only reports. firstSuccessVerified: benchmark-first-success.json cites evidence/run_output.txt at 71s, and the transcript shows that file already contained the successful 24-page HTTP 200 run before /usr/local/bin/benchmark-success was invoked.

Attempt record

  1. 5:51Answered from saved key: Context.dev API keyno wait
  2. —First success verified

Attempt artifacts saved.