Benchmark report

Task verified

4m 57s elapsed

The saved attempt met the documented success criterion.

Experience score

94/ 100

Agent-Ready
  • AInstall
  • SAuth
  • SExecute
  • ADocs
  • SCurrent
Experience measures friction in the tested task. The verdict says whether it succeeded.
Agent profile for this run: Install A, Auth S, Execute S, Docs A, Current SAInstallSAuthSExecuteADocsSCurrent
Agent profile · this run
Judge's note · what slowed the agent most
pip's PEP 668 refusal forced a venv detour, and the main messages docs page omitted the REST method/path, requiring an extra hop via llms.txt.

Where the time went

4:57 agent time
  • Research0:34
  • Attempt3:12
  • Verify1:11
4m 57sTotal elapsed
2:05Time to first success
0:00Waited on the owner

Judge reason

Saved code send_transactional_email.py calls the documented REST operation POST https://api.agentmail.to/v0/inboxes/{inbox_id}/messages/send with Authorization: Bearer from a key file (no SMTP/dashboard). outputs/01_send_message.json records the live request (to=bench-rcpt-20261109201018@agentmail.to, subject 'Your receipt for order #20261109201018', plain-text receipt body, labels) byte-matching the dict the script serializes and sends, HTTP 200 and raw body {"message_id":"<010001a1c1eede63-...@email.amazonses.com>","thread_id":"21a3fd71-..."} — a real SES id, not fabricated. console.log and the transcript tool result (step 64) print the same status and message_id. Sender was discovered via API: GET /v0/domains -> 200 with 0 verified custom domains and GET /v0/inboxes -> 200 (7 inboxes), so an existing inbox on the default managed domain agentmail.to was used. AgentMail documents no test recipient, so the recipient inbox was created for the run via POST /v0/inboxes (200, outputs/02) and deleted afterwards via DELETE /v0/inboxes/... -> 202 with empty body plus a confirming GET 404 (outputs/03), matching the documented delete semantics.

Attempt record

  1. 1:25Answered from saved key: AgentMail API keyno wait
  2. —First success verified

Attempt artifacts saved.