Benchmark report

Task verified

5m 6s elapsed

The saved attempt met the documented success criterion.

Experience score

86/ 100

Agent-Ready
  • SInstall
  • SAuth
  • AExecute
  • ADocs
  • SCurrent
Experience measures friction in the tested task. The verdict says whether it succeeded.
Agent profile for this run: Install S, Auth S, Execute A, Docs A, Current SSInstallSAuthAExecuteADocsSCurrent
Agent profile · this run
Judge's note · what slowed the agent most
The sandbox domain's authorized-recipient restriction surfaced only as a 403 at send time and wasn't in the send docs, forcing a search for GET /v5/sandbox/auth_recipients.

Where the time went

5:06 agent time
  • Research0:44
  • Attempt3:00
  • Verify1:21
5m 6sTotal elapsed
2:09Time to first success
0:00Waited on the owner

Judge reason

send_email.py POSTs multipart/form-data to https://api.mailgun.net/v3/sandbox427a2eb4e9714be9978af14ef418a11e.mailgun.org/messages with HTTP Basic auth (api:<key> read from the benchmark-provided secret file) and o:testmode=yes, Mailgun's documented test mode that the task explicitly permits with any recipient. Transcript idx 75/76 shows the live run: HTTP status 200, body {"id":"<20261105225422.75841b4a2bf7a08b@sandbox...mailgun.org>","message":"Queued. Thank you."}, exit=0. evidence/send_response.json records the same status plus live response headers (Account-Id 6a66ee21ed79844797da0896, X-Mailgun-Key-Id, X-Ratelimit-Remaining 50, Date Thu, 05 Nov 2026 22:54:22 GMT), so the response is real, not mocked. The saved request fields (from postmaster@<sandbox domain>, to rick@promptingco.com, subject 'Order #10492 confirmed', the plain-text body, o:testmode=yes) match the code and the printed stdout exactly. Sender was discovered through the API: GET /v4/domains on the US host returned the one active sandbox domain (domains_us.json) while the EU host returned total_count 0. A first send to postmaster@<sandbox> returned 403 ('add the address to your authorized recipients'); GET /v5/sandbox/auth_recipients showed rick@promptingco.com already activated, so no inbox or recipient was created and no deletion response is required. Exactly one email was accepted, via REST API rather than SMTP or dashboard, and the reported status and message id match the saved response.

Attempt record

  1. 1:22Answered from saved key: Mailgun API keyno wait
  2. —First success verified

Attempt artifacts saved.