Benchmark report

Task verified

10m 53s elapsed

The saved attempt met the documented success criterion.

Experience score

98/ 100

Agent-Ready
  • AInstall
  • SAuth
  • SExecute
  • SDocs
  • SCurrent
Experience measures friction in the tested task. The verdict says whether it succeeded.
Agent profile for this run: Install A, Auth S, Execute S, Docs S, Current SAInstallSAuthSExecuteSDocsSCurrent
Agent profile · this run
Judge's note · what slowed the agent most
The PEP 668 externally-managed-environment pip failure blocked the optional SDK install, forcing a stdlib urllib fallback; otherwise the run was linear.

Where the time went

10:53 agent time
  • Starting0:36
  • Research1:15
  • Attempt5:13
  • Verify3:49
10m 53sTotal elapsed
1:04Time to first success
0:00Waited on the owner

Judge reason

Transcript shows the agent fetched the official Parallel docs (advanced-search-settings.md, search-quickstart, api-reference/search/search.md), confirming advanced_settings.location is the only documented targeting field (ISO-3166 alpha-2, 'de' listed) and that V1SearchRequest/AdvancedSearchSettings (additionalProperties:false) define no language/locale/lang/hl/gl parameter — on stable /v1, with no beta/preview marking. It requested the key via benchmark-request (outcome 'provided'), then ran saved code /workspace/home/parallel-search-de/search_de.py (stdlib urllib POST https://api.parallel.ai/v1/search, x-api-key) carrying search_queries ['Datenbank Transaktionsisolation', ...] plus advanced_settings.location='de'. live_output.txt shows exit=0, HTTP 200, search_id search_b71ae998cc634678f150083c085ababc, warnings None, 10 results all with excerpts; response.json matches the documented schema (url/title/publish_date/excerpts, usage sku_search). At least 6 results are clearly German in title and excerpt (AWS de_de 'Definition von Isolationsstufen', learn.microsoft.com/de-de JDBC 'Grundlegendes zu Isolationsstufen', biancahoegel.de 'Isolation (Datenbank)', de.sqldat.com 'Transaktionsisolation in PostgreSQL', learn.microsoft.com/de-de locking guide, docwiki.embarcadero.com/.../de/), exceeding the 3-result bar. All 10 results were printed unfiltered and untranslated. The missing language parameter is explicitly named in result.json, which the success criteria permit. Minor: result.json's german_result_titles list labels 7 entries but transparently flags one as English — the verified German count is 6. stderr empty; no mocking or fabrication.

Attempt record

  1. 4:47Answered from saved key: Parallel API keyno wait
  2. —First success verified

Attempt artifacts saved.