Benchmark report
Task verified
10m 2s elapsed
The saved attempt met the documented success criterion.
94/ 100
Agent-Ready- AInstall
- SAuth
- SExecute
- ADocs
- SCurrent
Truncated quickstart and reference pages forced a detour: downloading the 404KB OpenAPI spec and grepping it to confirm userLocation and the absent language parameter.
Where the time went
10:03 agent time- Starting0:01
- Research0:46
- Attempt3:25
- Verify5:51
Judge reason
Verified live Exa search. Saved search_de.py posts to https://api.exa.ai/search with x-api-key from EXA_API_KEY and body {query:'Datenbank Transaktionsisolation', userLocation:'DE', type:'auto', numResults:10, contents.highlights:true}; request.json matches. Transcript line 50-51 shows the real run: exit=0, 'HTTP 200 POST https://api.exa.ai/search', 10 results. response.json is a genuine live payload (requestId 1e7b3067896fa03bc41216a67ebf2467, resolvedSearchType, searchTime 1029.8ms, costDollars 0.007) and its sha256 ee4750...d792 matches the one recorded in result.json. At least 6 of 10 results are German by title and highlight excerpt (de.wikipedia 'Isolation (Datenbank)', Oracle de docs, TUM kapitel10.pdf, postgresdb.de 'Transaktionsisolation', learn.microsoft.com/de-de, databricks.com/de), exceeding the 3-result threshold; all 10 results printed in API order with title, URL and excerpt, no local translation or filtering. The language-parameter gap is independently supported: the agent downloaded the official exa-spec.yaml (404674 bytes) and grep -niE '(language|lang|locale|hl|gl):' returned a single hit at line 7389, shown in the transcript to be a response field ('Publication language.') on a research-publication schema, not a SearchRequest property; the spec text for userLocation ('two-letter ISO 3166-1 country code ... The country localizes results') carries no beta/preview/gated marking. That satisfies the criterion allowing one documented targeting parameter plus naming the missing one, and MISSING_LANGUAGE_PARAM.md records the gap.
Attempt record
- 2:19Answered from saved key: Exa API keyno wait
- —First success verified
Attempt artifacts saved.