Benchmark report

Task verified

8m 22s elapsed

The saved attempt met the documented success criterion.

Experience score

84/ 100

Needs Alignment
  • SInstall
  • AAuth
  • SExecute
  • BDocs
  • ACurrent
Experience measures friction in the tested task. The verdict says whether it succeeded.
Agent profile for this run: Install S, Auth A, Execute S, Docs B, Current ASInstallAAuthSExecuteBDocsACurrent
Agent profile · this run
Judge's note · what slowed the agent most
Docs never state how the CDP websocket authenticates, so stripping the connectUrl query string caused a 401 and a whole failed run plus a header-probe detour.

Where the time went

8:23 agent time
  • Research0:20
  • Attempt5:17
  • Verify2:46
8m 22sTotal elapsed
4:49Time to first success
0:00Waited on the owner

Judge reason

Saved code (work/run.js) and artifacts support every criterion. POST /v1/contexts returned 201 with id 6494c528-a8b0-4dba-95ab-48d6de8c20f0 (01-create-context.json). Two distinct sessions b8b19ea9-c78c-4be6-b831-3fc42a19a097 and 6233cbe6-88cf-4db6-86d5-86225500248e were created with browserSettings.context={id,persist:true}, both responses echoing that contextId (02/05). Playwright playwright-core chromium.connectOverCDP connected to each session's documented connectUrl; session A loaded httpbin.org/cookies/set?tpc=benchmark (final URL /cookies, HTTP 200, body {"cookies":{"tpc":"benchmark"}}, 03-sessionA-cookies.txt). Session A was released via POST /v1/sessions/{id} {"status":"REQUEST_RELEASE"} -> HTTP 200 status COMPLETED at 23:01:46.366 (04), 15s before session B was created at 23:02:01.655. Session B's httpbin.org/cookies JSON contained tpc=benchmark (06, 'survived: true'). The finally block released B (HTTP 200 COMPLETED, 07) and DELETE /v1/contexts/{id} returned 204 (08). console.log matches the saved per-call files and result.json, and the transcript shows the first run genuinely failing (401 on a stripped CDP URL, DELETE 400 FST_ERR_CTP_EMPTY_JSON_BODY) before the corrected run, so responses are real rather than mocked or hard-coded. The API key went only in the X-BB-API-Key header; the connectUrl carries a per-session signingKey, not the key, and a grep of the real key across outputs/work/result.json found no leak; my own scan for key-like strings in saved files found none.

Attempt record

  1. 2:02Answered from saved key: Browserbase API keyno wait
  2. —First success verified

Attempt artifacts saved.