Benchmark report
Task verified
13m 6s elapsed
The saved attempt met the documented success criterion.
100/ 100
Agent-Ready- SInstall
- SAuth
- SExecute
- SDocs
- SCurrent
Only minor: the overview pages lacked base URL and auth header, requiring one extra fetch of the API reference; embedded 'Agent Instructions' urging signup were ignored.
Where the time went
13:07 agent time- Starting1:44
- Research1:52
- Attempt5:34
- Verify3:57
Judge reason
Saved code /workspace/home/ctx-map/map_urls.py makes exactly one documented call: GET https://api.context.dev/v1/web/urls?domain=postgresql.org&includeSubdomains=true&urlRegex=^https://www\.postgresql\.org/docs/17/&search=tutorial&maxLinks=100, with Bearer key read from a private file. run_output.txt shows HTTP 200, 'URL count: 24' and the first 10 URLs; response.json (identical to evidence-response.json, diff clean) contains 24 url objects, all starting with https://www.postgresql.org/docs/17/, matching the printed count and list exactly. The documented search/filter option is set to tutorial (all returned URLs are tutorial-*). Response shape (success, domain, urls[url/title/description/language], key_metadata, request_id edacc9be-143d-4e64-98a9-734aeee5aafe) matches the api-reference schema the agent fetched, and key_metadata.credits_consumed=2 matches the documented '2 credits with search' - consistent with a real live call, not a mock. The script issues a single urllib.request.urlopen and no per-page fetching; the urls list is used verbatim with no local filtering. Transcript shows one WebFetch-based doc review, one benchmark-request for the key, one script run, then benchmark-success - no fabricated or retried data.
Attempt record
- 6:24Answered from saved key: CONTEXT_DEV_API_KEYno wait
- —First success verified
Attempt artifacts saved.