Benchmark report

Task verified

5m 8s elapsed

The saved attempt met the documented success criterion.

Experience score

100/ 100

Agent-Ready
  • SInstall
  • SAuth
  • SExecute
  • SDocs
  • SCurrent
Experience measures friction in the tested task. The verdict says whether it succeeded.
Agent profile for this run: Install S, Auth S, Execute S, Docs S, Current SSInstallSAuthSExecuteSDocsSCurrent
Agent profile · this run
Judge's note · what slowed the agent most
Only minor: request fields (maxUrls/regex) differ from response read-back names (max_pages/url_pattern), needing careful doc reading before the first call.

Where the time went

5:09 agent time
  • Starting0:01
  • Research0:26
  • Attempt2:56
  • Verify1:46
5m 8sTotal elapsed
2:04Time to first success
0:00Waited on the owner

Judge reason

Live product-side crawl verified. crawl_tutorial.py submits one POST https://api.context.dev/v1/batch/submit with input.mode=crawl, source.url=https://www.postgresql.org/docs/17/tutorial.html, controls.maxUrls=25 and controls.regex=^https://www\.postgresql\.org/docs/17/tutorial set before submission (01_submit_request.json). Docs fetched in the transcript define controls.regex as 'URLs to include' (an include filter, not ranking) and map it to the response read-back url_pattern. 02_submit_response.json shows HTTP 200 and batch_71fa30cba9cf4a459e6420ebe7b09f5b; 03_poll_history/04_final_batch show polling to status=completed with crawl.max_pages=25, url_pattern echoed, input.reserved=25 reserved_is_ceiling=true, progress.succeeded=24, failed=0, timing 21:37:16->21:37:28. Results were paginated from GET /batch/{id}/results in the same script; the printed stdout captured in the transcript lists 24 distinct URLs with titles, all beginning with https://www.postgresql.org/docs/17/tutorial, matching 06_pages.json and printed_report.txt exactly; titles (e.g. 'Part I. Tutorial', '1.1. Installation', '3.5. Window Functions') match the real PostgreSQL 17 tutorial pages. No local filtering, no per-page fetching, no cap lowering, no mocking. The 277 KB 05_results_raw.json seen in the run's ls output is absent from the saved snapshot, but the derived page list, the live stdout and the server-side progress count of 24 corroborate the response.

Attempt record

  1. 1:26Answered from saved key: Context.dev API keyno wait
  2. —First success verified

Attempt artifacts saved.