Benchmark report

Task verified

8m 33s elapsed

The saved attempt met the documented success criterion.

Experience score

98/ 100

Agent-Ready
  • AInstall
  • SAuth
  • SExecute
  • SDocs
  • SCurrent
Experience measures friction in the tested task. The verdict says whether it succeeded.
Agent profile for this run: Install A, Auth S, Execute S, Docs S, Current SAInstallSAuthSExecuteSDocsSCurrent
Agent profile · this run
Judge's note · what slowed the agent most
The PEP 668 externally-managed-environment error on the first pip install, requiring a venv; everything else ran first try in ~87s.

Where the time went

8:34 agent time
  • Starting0:01
  • Research0:41
  • Attempt3:43
  • Verify4:08
8m 33sTotal elapsed
1:27Time to first success
0:00Waited on the owner

Judge reason

Saved evidence proves one live authenticated Parallel Search call. project/discover_urls.py posts a single client.search() (parallel-web 1.3.7, x-api-key read from the private key file) with search_queries ['tutorial'], mode 'advanced', advanced_settings.source_policy.include_domains ['www.postgresql.org/docs/17'], max_results 20; no fetching or local filtering appears (urls list is taken verbatim from response['results']). The transcript shows discover_urls.py executed exactly once (step 56) and only one search response id exists. evidence/search_response.json is a genuine API payload (search_id search_70982a4219ff1578371983469512ffb5, usage sku_search 1 + sku_additional_search_results 10, per-result titles/excerpts/publish_date 2026-08-13 citing PostgreSQL 17.11) with 20 results, 20/20 URLs under https://www.postgresql.org/docs/17/. evidence/run_output.txt prints 'URL count: 20' and the first 10 URLs, which match search_response.json and urls.json exactly. Saved docs (ref-source-policy.md line 31, lines 112-115) document include_domains as a hard domain/path-prefix allow list for the Search API, and /v1/search is GA (docs mark only /v1beta as legacy; no beta/preview/request-only marking).

Attempt record

  1. 2:44Answered from saved key: PARALLEL_API_KEYno wait
  2. —First success verified

Attempt artifacts saved.