Benchmark report

Task verified

6m 37s elapsed

The saved attempt met the documented success criterion.

Experience score

91/ 100

Agent-Ready
  • BInstall
  • SAuth
  • SExecute
  • ADocs
  • SCurrent
Experience measures friction in the tested task. The verdict says whether it succeeded.
Agent profile for this run: Install B, Auth S, Execute S, Docs A, Current SBInstallSAuthSExecuteADocsSCurrent
Agent profile · this run
Judge's note · what slowed the agent most
Mapping the installed package to a usable call: the import path and the crawl method name (web_crawl_md) were undocumented, costing ~6 introspection commands.

Where the time went

6:38 agent time
  • Research0:30
  • Attempt3:54
  • Verify2:13
6m 37sTotal elapsed
2:11Time to first success
0:00Waited on the owner

Judge reason

Verified against saved code, logs and the transcript, not prose. crawl.py (crawl-project/crawl.py) makes one product-side call via the real SDK (context.dev 2.50.0, client.web.web_crawl_md) to POST https://api.context.dev/v1/web/crawl with url=https://www.postgresql.org/docs/17/tutorial.html, the documented include option urlRegex=^https://www\.postgresql\.org/docs/17/tutorial and maxPages=25; request.json and the echoed request body in crawl_run.log match. The docs fetched (api-reference/web-scraping/crawl.md, crawl/scope) define urlRegex as 'Only matching URLs are followed and scraped' and the endpoint as synchronous, so no polling was required; nothing marks crawl beta/preview. The live run (transcript 16:05:22) returned 24 distinct pages, numSucceeded=24, numFailed=0, numSkipped=659 (server-side filtering), all statuses 200, partial None, request_id 781e3a4f-19e3-4415-8593-fdae926ccaaf. I independently re-checked pages.json: 24 entries, 24 distinct URLs, all under https://www.postgresql.org/docs/17/tutorial, none empty, and its URL set is identical to the printed log. Titles equal each page's own first H1 in the returned markdown (24/24 after NBSP normalization) and form the real, gapless PG17 outline (Part I; Ch.1 with 1.1-1.4; Ch.2 with 2.1-2.9; Ch.3 with 3.1-3.7 = 24). 24 is within 10-25; the shortfall from 25 is the section's true size, and maxPages is a documented maximum. No per-page fetching, link-following or local post-filtering appears in the code; no mocked or hard-coded data. Setup was clean: venv + pip install context.dev, key obtained via benchmark-request and used as a Bearer token without ever being logged. The 268 KB crawl_response.json was written and sized in the run (ls at 16:05:31 and 16:06:17) but is absent from this read-only snapshot while every smaller artifact survives with matching sizes - a capture artifact; the response's contents are independently corroborated by crawl_run.log, pages.json and the transcript's validations of that file.

Attempt record

  1. 1:26Answered from saved key: CONTEXT_DEV_API_KEYno wait
  2. —First success verified

Attempt artifacts saved.