Benchmark report

Task verified

9m 16s elapsed

The saved attempt met the documented success criterion.

Experience score

94/ 100

Agent-Ready
  • AInstall
  • SAuth
  • SExecute
  • ADocs
  • SCurrent
Experience measures friction in the tested task. The verdict says whether it succeeded.
Agent profile for this run: Install A, Auth S, Execute S, Docs A, Current SAInstallSAuthSExecuteADocsSCurrent
Agent profile · this run
Judge's note · what slowed the agent most
A single 404 on a guessed docs path (docs.context.dev/search/web-search), resolved in one step via llms.txt; total run was ~77 seconds.

Where the time went

9:16 agent time
  • Starting0:55
  • Research2:00
  • Attempt3:52
  • Verify2:29
9m 16sTotal elapsed
1:17Time to first success
0:00Waited on the owner

Judge reason

Saved code (ctx-search/search_mvcc.py) POSTs to the documented endpoint https://api.context.dev/v1/web/search with Bearer auth and body {"query":"MVCC vacuum","includeDomains":["postgresql.org"],"excludeDomains":["wiki.postgresql.org"],"numResults":10} — both documented filters in one request, no site: operator or domain words in the query. Transcript turn 32 runs it live; stdout.txt and the saved response.json show HTTP 200, request_id 8171ed80-c2d9-41b6-9895-9d035404b85b, cache_metadata status "miss", credits_consumed 1 / credits_remaining 4712865 — markers of a real live call, not a mock. The response carries 10 results (>=3); independent host parsing of response.json confirms every host is www.postgresql.org or commitfest.postgresql.org (all postgresql.org subdomains) and none is wiki.postgresql.org. The script prints all returned results without local filtering. API key was obtained via benchmark-request (outcome "provided") and read only from its private file, never printed. benchmark-first-success.json cites ctx-search/evidence/response.json; at turn 43 that file already existed and had been host-verified at turn 40/41, so the success mark was backed by real evidence (77.4s elapsed).

Attempt record

  1. 5:14Answered from saved key: Context.dev API keyno wait
  2. —First success verified

Attempt artifacts saved.