Benchmark report

Task verified

12m 30s elapsed

The saved attempt met the documented success criterion.

Experience score

98/ 100

Agent-Ready
  • AInstall
  • SAuth
  • SExecute
  • SDocs
  • SCurrent
Experience measures friction in the tested task. The verdict says whether it succeeded.
Agent profile for this run: Install A, Auth S, Execute S, Docs S, Current SAInstallSAuthSExecuteSDocsSCurrent
Agent profile · this run
Judge's note · what slowed the agent most
Only the upfront doc reading (six pages) before coding; the whole lifecycle completed in ~116s with no errors or retries.

Where the time went

12:31 agent time
  • Starting1:26
  • Research1:58
  • Attempt5:55
  • Verify3:12
12m 30sTotal elapsed
1:56Time to first success
0:00Waited on the owner

Judge reason

Saved runnable code (firecrawl-monitor-task/monitor_lifecycle.py, stdlib urllib against https://api.firecrawl.dev/v2) and the full raw transcript (evidence/api_transcript.json, run_console.log) show a real authenticated lifecycle: POST /v2/monitor -> 200 with id 01a1c75b-64a3-73c8-9868-5ea6152e2868 and schedule cron '*/5 * * * *' (docs-confirmed 5-minute minimum); GET /v2/monitor/{id} -> 200 reading back targets[0].urls = https://www.postgresql.org/docs/17/release-17.html and the same cron/UTC schedule; documented manual trigger POST /v2/monitor/{id}/run -> 200 (check 01a1c75b-65a9-75b4-8027-9bc43df5b943, trigger=manual) polled to status=completed with a real scraped page (statusCode 200, title 'PostgreSQL: Documentation: 17: E.12. Release 17', 1 credit billed) and confirmed again via GET /checks; DELETE /v2/monitor/{id} -> 200 {"success": true} issued from a finally block (cleanup on failure), confirmed by GET -> 404 {"error": "Monitor not found"}. No mocks or hard-coded responses; docs were checked for beta/preview markings and none found. Setup was clean: key requested once via benchmark-request, smoke test GET /v2/monitor returned 200 on first try.

Attempt record

  1. 5:58Answered from saved key: FIRECRAWL_API_KEYno wait
  2. —First success verified

Attempt artifacts saved.