Benchmark report

Task verified

10m 10s elapsed

The saved attempt met the documented success criterion.

Experience score

93/ 100

Agent-Ready
  • SInstall
  • AAuth
  • SExecute
  • SDocs
  • SCurrent
Experience measures friction in the tested task. The verdict says whether it succeeded.
Agent profile for this run: Install S, Auth A, Execute S, Docs S, Current SSInstallAAuthSExecuteSDocsSCurrent
Agent profile · this run
Judge's note · what slowed the agent most
Its own HTTP-only socket guard blocked the sandbox's mandatory HTTP CONNECT egress proxy, failing the first run and costing a patch cycle.

Where the time went

10:10 agent time
  • Research0:30
  • Attempt6:41
  • Verify3:00
10m 10sTotal elapsed
4:33Time to first success
0:00Waited on the owner

Judge reason

Verified against saved artifacts, not prose. work/app/xata-http-sql.mjs uses neon() from @neondatabase/serverless v1.2.0 (HTTP fetch mode, no Pool/Client); the run (transcript idx 132, work/out/sql/run-stdout.txt) shows 4 live POSTs to https://api.us-east-1.xata.tech/sql all 200 with distinct x-request-ids, and server-reported PostgreSQL 17.11 plus server_time 2026-11-06T00:31:44Z, so nothing is mocked. Raw responses sql-1..3 show CREATE, INSERT rowCount 2 and SELECT rowCount 2 returning id=1 widget 7 and id=2 gadget 42, exactly matching printed-rows.json and stdout. The script wraps net.connect/net.createConnection/tls.connect and stubs WebSocket and http.request before loading the driver; http-request-log.json records only the sandbox HTTP CONNECT proxy hop and tls.connect api.us-east-1.xata.tech:443 - no 5432, no WebSocket (an earlier run, idx 116, actually aborted on that guard, proving it is live). Project tpcbench-438b0312d024 and branch main were created through https://api.xata.tech. Cleanup is complete: branch DELETE 204 (headers file), GET 404 'branch ... not found', branch list [], project DELETE 204, GET 404 'project ... not found', and 15-list-projects-after-delete.json contains zero tpcbench matches. A grep of every saved file finds no API key and no password; the only connection string present is redacted to xata:***. One hygiene slip: the create-branch curl output echoed the live connection string once into the transcript (idx 67) before the agent redacted the saved file - the graded artifacts and the deleted branch leave nothing exposed.

Attempt record

  1. 1:40Answered from saved key: XATA_API_KEYno wait
  2. —First success verified

Attempt artifacts saved.