Benchmark report

Task verified

4m 44s elapsed

The saved attempt met the documented success criterion.

Experience score

86/ 100

Agent-Ready
  • AInstall
  • AAuth
  • SExecute
  • ADocs
  • SCurrent
Experience measures friction in the tested task. The verdict says whether it succeeded.
Agent profile for this run: Install A, Auth A, Execute S, Docs A, Current SAInstallAAuthSExecuteADocsSCurrent
Agent profile · this run
Judge's note · what slowed the agent most
The account's 3-inbox plan cap was already full, forcing the agent to delete its own scratch sender inbox and rewrite the setup script before the recipient could be created.

Where the time went

4:44 agent time
  • Research0:18
  • Attempt3:21
  • Verify1:05
4m 44sTotal elapsed
2:24Time to first success
0:00Waited on the owner

Judge reason

Saved code (agentmail_client.py + send_email.py) makes a live Bearer-authenticated POST to AgentMail's documented send operation, https://api.agentmail.to/v0/inboxes/evt-test-1793254442%40agentmail.to/messages/send (docs page fetched in-transcript at line 16 confirms the path, Bearer auth, and the 200 shape {message_id, thread_id}). AgentMail documents no public test recipient, so the fallback applied: the sole recipient bench-sink-20261029@agentmail.to was created for this run via POST /v0/inboxes (create_inbox_recipient_response.json, HTTP 200), and the sender was discovered through the API with GET /v0/inboxes (list_inboxes_response.json). send_message_request.json matches the literals in send_email.py exactly (subject 'Your order AM-4817 has shipped' and the plain-text body). The real run printed HTTP 200 with message_id <010001a1864f29ef-fc709528-7e29-4cf6-b458-41d47e32c576-000000@email.amazonses.com> and thread_id 709ecab9-18a7-4316-8235-a8cde7053ccf, identical to send_message_response.json/send_message_status.txt and to transcript line 82. Independently corroborated by a separate curl GET /v0/inboxes/bench-sink-20261029%40agentmail.to/messages (transcript line 90) returning count 1 with the same message_id, matching subject/body preview, from the sender inbox, spf/dkim/dmarc pass. Exactly one email, sent over the REST API, not SMTP or a dashboard; no mocking or hard-coded response.

Attempt record

  1. 0:29Answered from saved key: AgentMail API keyno wait
  2. —First success verified

Attempt artifacts saved.