Benchmark report
Task verified
5m 8s elapsed
The saved attempt met the documented success criterion.
100/ 100
Agent-Ready- SInstall
- SAuth
- SExecute
- SDocs
- SCurrent
Only minor: request fields (maxUrls/regex) differ from response read-back names (max_pages/url_pattern), needing careful doc reading before the first call.
Where the time went
5:09 agent time- Starting0:01
- Research0:26
- Attempt2:56
- Verify1:46
Judge reason
Live product-side crawl verified. crawl_tutorial.py submits one POST https://api.context.dev/v1/batch/submit with input.mode=crawl, source.url=https://www.postgresql.org/docs/17/tutorial.html, controls.maxUrls=25 and controls.regex=^https://www\.postgresql\.org/docs/17/tutorial set before submission (01_submit_request.json). Docs fetched in the transcript define controls.regex as 'URLs to include' (an include filter, not ranking) and map it to the response read-back url_pattern. 02_submit_response.json shows HTTP 200 and batch_71fa30cba9cf4a459e6420ebe7b09f5b; 03_poll_history/04_final_batch show polling to status=completed with crawl.max_pages=25, url_pattern echoed, input.reserved=25 reserved_is_ceiling=true, progress.succeeded=24, failed=0, timing 21:37:16->21:37:28. Results were paginated from GET /batch/{id}/results in the same script; the printed stdout captured in the transcript lists 24 distinct URLs with titles, all beginning with https://www.postgresql.org/docs/17/tutorial, matching 06_pages.json and printed_report.txt exactly; titles (e.g. 'Part I. Tutorial', '1.1. Installation', '3.5. Window Functions') match the real PostgreSQL 17 tutorial pages. No local filtering, no per-page fetching, no cap lowering, no mocking. The 277 KB 05_results_raw.json seen in the run's ls output is absent from the saved snapshot, but the derived page list, the live stdout and the server-side progress count of 24 corroborate the response.
Attempt record
- 1:26Answered from saved key: Context.dev API keyno wait
- —First success verified
Attempt artifacts saved.