Benchmark report

Task verified

16m 28s elapsed

The saved attempt met the documented success criterion.

Experience score

95/ 100

Agent-Ready
  • SInstall
  • SAuth
  • SExecute
  • ADocs
  • ACurrent
Experience measures friction in the tested task. The verdict says whether it succeeded.
Agent profile for this run: Install S, Auth S, Execute S, Docs A, Current ASInstallSAuthSExecuteADocsACurrent
Agent profile · this run
Judge's note · what slowed the agent most
A 404 on a guessed docs path and the undocumented 'superseded' manual-run dedup, which prompted a second full lifecycle run to capture a completed run result.

Where the time went

16:29 agent time
  • Starting0:55
  • Research1:57
  • Attempt6:19
  • Verify7:18
16m 28sTotal elapsed
2:03Time to first success
0:00Waited on the owner

Judge reason

Saved code project/monitor_lifecycle.py (stdlib urllib, key read from a private file, cleanup in a finally block) drove the documented Context.dev Monitors REST API live. evidence/api-transcript.json + run-console.log show POST https://api.context.dev/v1/monitors -> HTTP 201 id mon_f2e803cd34c94735a2368c2739b274e9 with target url https://www.postgresql.org/docs/17/release-17.html and schedule {interval,10,minutes}; GET /v1/monitors/{id} -> 200 reading back the same url and schedule (url_matches/schedule_matches true); the documented manual trigger POST /v1/monitors/{id}/run -> 202 run_8b1f607..., and GET /v1/monitors/{id}/runs/{run_id} -> 200 read the run back (terminal status skipped, skip_reason 'superseded'); DELETE -> 200 {"deleted": true} and a follow-up GET -> 404 NOT_FOUND. A second cycle (api-transcript-v2.json, mon_c4389babae6b4864b44283d38eb4e547) additionally read back a completed baseline run (status completed, baseline_created true, credits_charged 1) and repeated the confirmed delete + 404. Responses are server-side, not mocked: unique X-Request-Id headers echoed in each body, X-RateLimit-Remaining decrementing 999->989 across both processes, and credits_remaining falling monotonically 4711857 -> 4711684. 10 minutes is the shortest interval the official OpenAPI spec allows ('interval (frequency x unit) must be at least 10 minutes'), and the spec check printed beta=none/depr=false for every Monitors operation, so the no-beta stop condition did not apply. Both monitors created were deleted; nothing left behind.

Attempt record

  1. 6:16Answered from saved key: Context.dev API keyno wait
  2. —First success verified

Attempt artifacts saved.