Benchmark report

Task verified

10m 2s elapsed

The saved attempt met the documented success criterion.

Experience score

94/ 100

Agent-Ready
  • AInstall
  • SAuth
  • SExecute
  • ADocs
  • SCurrent
Experience measures friction in the tested task. The verdict says whether it succeeded.
Agent profile for this run: Install A, Auth S, Execute S, Docs A, Current SAInstallSAuthSExecuteADocsSCurrent
Agent profile · this run
Judge's note · what slowed the agent most
Truncated quickstart and reference pages forced a detour: downloading the 404KB OpenAPI spec and grepping it to confirm userLocation and the absent language parameter.

Where the time went

10:03 agent time
  • Starting0:01
  • Research0:46
  • Attempt3:25
  • Verify5:51
10m 2sTotal elapsed
1:28Time to first success
0:00Waited on the owner

Judge reason

Verified live Exa search. Saved search_de.py posts to https://api.exa.ai/search with x-api-key from EXA_API_KEY and body {query:'Datenbank Transaktionsisolation', userLocation:'DE', type:'auto', numResults:10, contents.highlights:true}; request.json matches. Transcript line 50-51 shows the real run: exit=0, 'HTTP 200 POST https://api.exa.ai/search', 10 results. response.json is a genuine live payload (requestId 1e7b3067896fa03bc41216a67ebf2467, resolvedSearchType, searchTime 1029.8ms, costDollars 0.007) and its sha256 ee4750...d792 matches the one recorded in result.json. At least 6 of 10 results are German by title and highlight excerpt (de.wikipedia 'Isolation (Datenbank)', Oracle de docs, TUM kapitel10.pdf, postgresdb.de 'Transaktionsisolation', learn.microsoft.com/de-de, databricks.com/de), exceeding the 3-result threshold; all 10 results printed in API order with title, URL and excerpt, no local translation or filtering. The language-parameter gap is independently supported: the agent downloaded the official exa-spec.yaml (404674 bytes) and grep -niE '(language|lang|locale|hl|gl):' returned a single hit at line 7389, shown in the transcript to be a response field ('Publication language.') on a research-publication schema, not a SearchRequest property; the spec text for userLocation ('two-letter ISO 3166-1 country code ... The country localizes results') carries no beta/preview/gated marking. That satisfies the criterion allowing one documented targeting parameter plus naming the missing one, and MISSING_LANGUAGE_PARAM.md records the gap.

Attempt record

  1. 2:19Answered from saved key: Exa API keyno wait
  2. —First success verified

Attempt artifacts saved.