Benchmark report

Task verified

9m 43s elapsed

The saved attempt met the documented success criterion.

Experience score

100/ 100

Agent-Ready
  • SInstall
  • SAuth
  • SExecute
  • SDocs
  • SCurrent
Experience measures friction in the tested task. The verdict says whether it succeeded.
Agent profile for this run: Install S, Auth S, Execute S, Docs S, Current SSInstallSAuthSExecuteSDocsSCurrent
Agent profile · this run
Judge's note · what slowed the agent most
Minimal: WebFetch returned only summarized doc pages, so the agent had to curl the raw .md OpenAPI spec to get exact request/response schemas.

Where the time went

9:44 agent time
  • Starting0:01
  • Research0:49
  • Attempt6:51
  • Verify2:02
9m 43sTotal elapsed
1:22Time to first success
0:00Waited on the owner

Judge reason

Saved code evidence/ask_context.py POSTs the exact question to the documented answer operation POST https://api.context.dev/v1/web/answers (mode=fast, task, json_format) and writes the raw response; it contains no mock or hard-coded result. evidence/response.json records a live HTTP 200 with server-only artifacts (X-Request-Id b999591c-0b9a-413e-97c2-9d7ea5e9f35e, Cf-Ray a58807ce8b458736-PDX, X-Credits-Used 10 matching the doc's 10-credit fast mode, credits_remaining 4712278), json_content.answer 'READ COMMITTED' and sources/citations ['https://www.postgresql.org/docs/current/sql-set-transaction.html'] — an absolute HTTPS URL returned by the product. stdout.txt and transcript idx 45/51 match the saved response exactly. The fetched spec (evidence/answers-doc-raw.md) shows no beta/preview/request-only marking on /web/answers, so the stop clause does not apply. Auth used a Bearer key obtained via benchmark-request; no key material leaked into artifacts (grep check returned 0).

Attempt record

  1. 4:28Answered from saved key: CONTEXT_DEV_API_KEYno wait
  2. —First success verified

Attempt artifacts saved.