Benchmark report

Task verified

12m 15s elapsed

The saved attempt met the documented success criterion.

Experience score

94/ 100

Agent-Ready
  • AInstall
  • SAuth
  • SExecute
  • ADocs
  • SCurrent
Experience measures friction in the tested task. The verdict says whether it succeeded.
Agent profile for this run: Install A, Auth S, Execute S, Docs A, Current SAInstallSAuthSExecuteADocsSCurrent
Agent profile · this run
Judge's note · what slowed the agent most
PEP 668 blocked the Python SDK install, forcing a stdlib urllib rewrite; filters.md's incomplete country entry also cost an extra API-reference fetch.

Where the time went

12:16 agent time
  • Starting0:46
  • Research1:26
  • Attempt6:47
  • Verify3:16
12m 15sTotal elapsed
1:54Time to first success
0:00Waited on the owner

Judge reason

Saved code context-search-de/search_de.py POSTs to the documented endpoint https://api.context.dev/v1/web/search with Bearer auth and body {"query":"Datenbank Transaktionsisolation","country":"de","languages":["de"],"numResults":10} — both targeting parameters the docs define (transcript entries 8/17/26 confirm filters.md and the API reference document `country` as a lowercase ISO 3166-1 code mapping to Google's gl and `languages` as a nonempty code array, neither marked beta/preview/gated, and no separate `location` parameter exists, which the agent explicitly named). The live run (transcript entry 51/52) produced response.json with API-side artifacts that cannot be mocked plausibly: request_id 94d846dc-ead4-4507-bcb3-5de4add87b21, cache_metadata status "miss", credits_consumed 1, credits_remaining 4713669, plus markdown/highlights code "NOT_REQUESTED" matching the documented response schema. The response holds 9 results, far above the 3 required, with German titles and excerpts (de.wikipedia.org "Isolation (Datenbank)", datenbanken-verstehen.de "Isolationsgrad - Isolation Level", db-berater.de "Isolationsstufe SERIALIZABLE im Detail", boddenberg.de "Transaktionsisolation SQL Server", learn.microsoft.com/de-de). stdout.txt prints title, URL and excerpt for every result in API order; the code contains no local translation, filtering or re-ranking — targeting came solely from the request parameters. API key was obtained via benchmark-request (outcome "provided") and read from a private file, never printed.

Attempt record

  1. 5:19Answered from saved key: CONTEXT_DEV_API_KEYno wait
  2. —First success verified

Attempt artifacts saved.