Benchmark report

Blocked by a prerequisite

3m 31s elapsed

An external prerequisite prevented the documented task from completing. Benchmark again

Experience score

64/ 100

Needs Alignment
  • AInstall
  • SAuth
  • CExecute
  • ADocs
  • SCurrent
Experience measures friction in the tested task. The verdict says whether it succeeded.
Agent profile for this run: Install A, Auth S, Execute C, Docs A, Current SAInstallSAuthCExecuteADocsSCurrent
Agent profile · this run
Judge's note · what slowed the agent most
A plan domain limit of 0 returned 403 on domain creation, and re-requesting a domain-capable key yielded the identical credential, halting the task at the first API step.

Where the time went

3:31 agent time
  • Research0:15
  • Attempt2:03
  • Verify1:13
3m 31sTotal elapsed
—Time to first success
0:00Waited on the owner

Judge reason

Setup was completed: the agent found the official docs (docs.agentmail.to/api-reference/domains/create and /domains/verify), obtained an API key through benchmark-request, and proved working authentication against the live API - auth_probe.json/probe_output.txt show GET /v0/domains -> HTTP 200 {"count":0,"domains":[]} and GET /v0/inboxes?limit=1 -> HTTP 200 with a real inbox. verify_domain.py is a complete runnable end-to-end script (create -> GET records -> POST verify -> GET status -> DELETE -> confirm). However none of the success criteria were met: the create call for tpc-bench-3938013b.example.com returned HTTP 403 limit_exceeded, so no domain_id exists, no DNS records were returned (result.json dns_records_returned: null), no verification call was made, no status was read (final_status: null) and no deletion response exists (deleted: false). The agent correctly refrained from claiming verified or fabricating responses, did not call benchmark-success, and reported outcome "blocked". The failure is a plan-level quota stated by the API itself, not a failed command or absent evidence.

Attempt record

  1. 1:38Answered from saved key: AgentMail API key for an account whose plan allows at least 1 custom domainno wait
  2. 2:18Blocked by a prerequisiteAgentMail account behind the supplied API key has a custom-domain plan limit of 0: POST https://api.agentmail.to/v0/domains returned HTTP 403 {"code":"limit_exceeded","resource":"domain","limit":0,"fix":"Upgrade at https://console.agentmail.to/dashboard/upgrade"} (api_transcript.json, run_output.txt). Raising the quota is a paid dashboard action with no API route; a second benchmark-request returned the byte-identical key (cmp -> SAME_AS_BEFORE).

Attempt artifacts saved.