Benchmark report
Task verified
5m 54s elapsed
The saved attempt met the documented success criterion.
86/ 100
Agent-Ready- AInstall
- AAuth
- SExecute
- ADocs
- SCurrent
Docs omitted the auth header name and a Playwright-over-CDP snippet on the create page, forcing an extra OpenAPI spec fetch and page hops.
Where the time went
5:54 agent time- Research0:18
- Attempt3:56
- Verify1:40
Judge reason
Transcript shows a real end-to-end run (the saved /workspace/home snapshot is truncated to venv files only, so evidence is the recorded commands/outputs). Agent installed kernel 0.139.0 + playwright 1.64.1 in a venv, requested the key via benchmark-request, and ran kernel_session.py whose code (captured verbatim in the Write call) calls the documented ops: kernel.browsers.with_raw_response.create(headless=True) -> HTTP 200 with session_id n1csd73h9x9ji9pun2uywqjr, cdp_ws_url wss://prod-jfk-hypeman-25.kernel.sh:8443/browser/cdp?<redacted>, created_at 2026-11-03T22:08:39Z; playwright chromium.connect_over_cdp(cdp_ws_url, headers={'Authorization': 'Bearer ...'}) then page.goto example.com printing 'page url: https://example.com/' and 'page title: Example Domain'; delete_by_id in a finally block returning HTTP 204 (spec.json documents 204 'Browser session deleted successfully'). Post-run GET /browsers/<id> returned 404 not_found, confirming real termination. Saved artifacts create_response.json, delete_response.json, run_summary.json, console_output.txt, commands.md and result.json were printed and match the console values; grep -rF of the key across /tmp/benchmark-home found no match, and the key was sent only in the Authorization header, never in a URL/query. Nothing mocked or hard-coded.
Attempt record
- 1:35Answered from saved key: KERNEL_API_KEYno wait
- —First success verified
Attempt artifacts saved.