Benchmark report
Task verified
8m 22s elapsed
The saved attempt met the documented success criterion.
84/ 100
Needs Alignment- SInstall
- AAuth
- SExecute
- BDocs
- ACurrent
Docs never state how the CDP websocket authenticates, so stripping the connectUrl query string caused a 401 and a whole failed run plus a header-probe detour.
Where the time went
8:23 agent time- Research0:20
- Attempt5:17
- Verify2:46
Judge reason
Saved code (work/run.js) and artifacts support every criterion. POST /v1/contexts returned 201 with id 6494c528-a8b0-4dba-95ab-48d6de8c20f0 (01-create-context.json). Two distinct sessions b8b19ea9-c78c-4be6-b831-3fc42a19a097 and 6233cbe6-88cf-4db6-86d5-86225500248e were created with browserSettings.context={id,persist:true}, both responses echoing that contextId (02/05). Playwright playwright-core chromium.connectOverCDP connected to each session's documented connectUrl; session A loaded httpbin.org/cookies/set?tpc=benchmark (final URL /cookies, HTTP 200, body {"cookies":{"tpc":"benchmark"}}, 03-sessionA-cookies.txt). Session A was released via POST /v1/sessions/{id} {"status":"REQUEST_RELEASE"} -> HTTP 200 status COMPLETED at 23:01:46.366 (04), 15s before session B was created at 23:02:01.655. Session B's httpbin.org/cookies JSON contained tpc=benchmark (06, 'survived: true'). The finally block released B (HTTP 200 COMPLETED, 07) and DELETE /v1/contexts/{id} returned 204 (08). console.log matches the saved per-call files and result.json, and the transcript shows the first run genuinely failing (401 on a stripped CDP URL, DELETE 400 FST_ERR_CTP_EMPTY_JSON_BODY) before the corrected run, so responses are real rather than mocked or hard-coded. The API key went only in the X-BB-API-Key header; the connectUrl carries a per-session signingKey, not the key, and a grep of the real key across outputs/work/result.json found no leak; my own scan for key-like strings in saved files found none.
Attempt record
- 2:02Answered from saved key: Browserbase API keyno wait
- —First success verified
Attempt artifacts saved.