Benchmark report

Task verified

12m 30s elapsed

The saved attempt met the documented success criterion.

Experience score

98/ 100

Agent-Ready
  • AInstall
  • SAuth
  • SExecute
  • SDocs
  • SCurrent
Experience measures friction in the tested task. The verdict says whether it succeeded.
Agent profile for this run: Install A, Auth S, Execute S, Docs S, Current SAInstallSAuthSExecuteSDocsSCurrent
Agent profile · this run
Judge's note · what slowed the agent most
Nothing material: the only cost was grepping the 101KB OpenAPI reference to confirm includeDomains/excludeDomains names and absence of beta marking; first success at 51s.

Where the time went

12:30 agent time
  • Starting0:01
  • Research0:16
  • Attempt5:27
  • Verify6:47
12m 30sTotal elapsed
0:51Time to first success
0:00Waited on the owner

Judge reason

Saved code exa-domain-search/search_domains.py posts to https://api.exa.ai/search with x-api-key and body {"query":"MVCC vacuum","type":"auto","numResults":10,"includeDomains":["postgresql.org"],"excludeDomains":["wiki.postgresql.org"]} - the documented GA parameters, no site: operator or domain words in the query, and no local filtering (the script only prints assertions). Transcript entry 31/32 shows the live run returning HTTP 200, EXIT=0, and response.json holds a genuine Exa payload (requestId 83ce49ccd0a3ae99c1ec012533733a44, searchTime 1793.8, costDollars 0.007) with 10 results, every host www.postgresql.org (a subdomain of postgresql.org) and none wiki.postgresql.org - exceeding the >=3 result bar. Key was obtained via benchmark-request (outcome provided) and read from a private file, never embedded. Docs check at transcript entry 14 found no beta/preview marking on includeDomains/excludeDomains.

Attempt record

  1. 4:16Answered from saved key: Exa API keyno wait
  2. —First success verified

Attempt artifacts saved.