Benchmark report

Task verified

16m 17s elapsed

The saved attempt met the documented success criterion.

Experience score

68/ 100

Needs Alignment
  • SInstall
  • BAuth
  • AExecute
  • ADocs
  • CCurrent
Experience measures friction in the tested task. The verdict says whether it succeeded.
Agent profile for this run: Install S, Auth B, Execute A, Docs A, Current CSInstallBAuthAExecuteADocsCCurrent
Agent profile · this run
Judge's note · what slowed the agent most
Steel's CDP socket rejects the API key in any header (only ?apiKey= or connect token), costing ~5 min of probing; then docs' nonexistent profiles.delete cost another ~4 min.

Where the time went

16:18 agent time
  • Research0:32
  • Attempt12:45
  • Verify3:01
16m 17sTotal elapsed
9:29Time to first success
0:00Waited on the owner

Judge reason

Saved code (work/steel_profile_cookie.mjs) and its real run output prove the whole flow. Session A was created with client.sessions.create({persistProfile:true}) which minted profile e90c3e56-7258-49dc-a9c0-83e1e6ae9ebd (the documented profile-creation path per docs/profiles-overview.md), id printed in run.log and saved in evidence/01. Playwright connectOverCDP to session A's websocketUrl loaded httpbin.org/cookies/set?tpc=benchmark (evidence/03, HTTP 200, body {"cookies":{"tpc":"benchmark"}}). sessions.release(A) returned {"success":true,...} (evidence/04) and the profile was polled UPLOADING -> READY (evidence/05) BEFORE session B was created at 23:37:17 (evidence/06, id ba43e369-..., distinct, same profileId). Session B over CDP returned {"cookies":{"tpc":"benchmark"}} from httpbin.org/cookies (evidence/07) - the cookie survived. finally block released both sessions and GET /v1/sessions/{id} shows both "released" (evidence/08). API key sent only in the steel-api-key header; the websocketUrl carries only Steel's own session connect token, which is redacted in every artifact; my own grep of /workspace/home found no key material and the agent's in-run grep -rlF over the home dir printed "no api key in any saved file". Printed run.log values match the saved JSON exactly. One clause could not be met: profile deletion. docs.steel.dev/cookbook/profiles names client.profiles.delete(id), but the call issued from the finally block returned HTTP 421 "No route defined for DELETE /v1/profiles/{id}" (evidence/08), and probes show CORS allow-methods GET,HEAD,PATCH,OPTIONS for /v1/profiles/{id} (evidence/09). I independently confirmed this: steel-sdk 0.18.0 (npm latest) src/resources/profiles.ts exposes only create, update, list, get - no delete. That is a product/doc defect, not an agent shortfall; the documented call was still issued in finally and its exact response saved, so the demonstrated core result stands.

Attempt record

  1. 3:07Answered from saved key: STEEL_API_KEYno wait
  2. —First success verified

Attempt artifacts saved.