Benchmark report

Task verified

13m 6s elapsed

The saved attempt met the documented success criterion.

Experience score

100/ 100

Agent-Ready
  • SInstall
  • SAuth
  • SExecute
  • SDocs
  • SCurrent
Experience measures friction in the tested task. The verdict says whether it succeeded.
Agent profile for this run: Install S, Auth S, Execute S, Docs S, Current SSInstallSAuthSExecuteSDocsSCurrent
Agent profile · this run
Judge's note · what slowed the agent most
Only minor: the overview pages lacked base URL and auth header, requiring one extra fetch of the API reference; embedded 'Agent Instructions' urging signup were ignored.

Where the time went

13:07 agent time
  • Starting1:44
  • Research1:52
  • Attempt5:34
  • Verify3:57
13m 6sTotal elapsed
1:09Time to first success
0:00Waited on the owner

Judge reason

Saved code /workspace/home/ctx-map/map_urls.py makes exactly one documented call: GET https://api.context.dev/v1/web/urls?domain=postgresql.org&includeSubdomains=true&urlRegex=^https://www\.postgresql\.org/docs/17/&search=tutorial&maxLinks=100, with Bearer key read from a private file. run_output.txt shows HTTP 200, 'URL count: 24' and the first 10 URLs; response.json (identical to evidence-response.json, diff clean) contains 24 url objects, all starting with https://www.postgresql.org/docs/17/, matching the printed count and list exactly. The documented search/filter option is set to tutorial (all returned URLs are tutorial-*). Response shape (success, domain, urls[url/title/description/language], key_metadata, request_id edacc9be-143d-4e64-98a9-734aeee5aafe) matches the api-reference schema the agent fetched, and key_metadata.credits_consumed=2 matches the documented '2 credits with search' - consistent with a real live call, not a mock. The script issues a single urllib.request.urlopen and no per-page fetching; the urls list is used verbatim with no local filtering. Transcript shows one WebFetch-based doc review, one benchmark-request for the key, one script run, then benchmark-success - no fabricated or retried data.

Attempt record

  1. 6:24Answered from saved key: CONTEXT_DEV_API_KEYno wait
  2. —First success verified

Attempt artifacts saved.