Benchmark report
Task verified
9m 16s elapsed
The saved attempt met the documented success criterion.
94/ 100
Agent-Ready- AInstall
- SAuth
- SExecute
- ADocs
- SCurrent
A single 404 on a guessed docs path (docs.context.dev/search/web-search), resolved in one step via llms.txt; total run was ~77 seconds.
Where the time went
9:16 agent time- Starting0:55
- Research2:00
- Attempt3:52
- Verify2:29
Judge reason
Saved code (ctx-search/search_mvcc.py) POSTs to the documented endpoint https://api.context.dev/v1/web/search with Bearer auth and body {"query":"MVCC vacuum","includeDomains":["postgresql.org"],"excludeDomains":["wiki.postgresql.org"],"numResults":10} — both documented filters in one request, no site: operator or domain words in the query. Transcript turn 32 runs it live; stdout.txt and the saved response.json show HTTP 200, request_id 8171ed80-c2d9-41b6-9895-9d035404b85b, cache_metadata status "miss", credits_consumed 1 / credits_remaining 4712865 — markers of a real live call, not a mock. The response carries 10 results (>=3); independent host parsing of response.json confirms every host is www.postgresql.org or commitfest.postgresql.org (all postgresql.org subdomains) and none is wiki.postgresql.org. The script prints all returned results without local filtering. API key was obtained via benchmark-request (outcome "provided") and read only from its private file, never printed. benchmark-first-success.json cites ctx-search/evidence/response.json; at turn 43 that file already existed and had been host-verified at turn 40/41, so the success mark was backed by real evidence (77.4s elapsed).
Attempt record
- 5:14Answered from saved key: Context.dev API keyno wait
- —First success verified
Attempt artifacts saved.