Benchmark report

Task verified

4m 30s elapsed

The saved attempt met the documented success criterion.

Experience score

90/ 100

Agent-Ready
  • SInstall
  • SAuth
  • AExecute
  • SDocs
  • SCurrent
Experience measures friction in the tested task. The verdict says whether it succeeded.
Agent profile for this run: Install S, Auth S, Execute A, Docs S, Current SSInstallSAuthAExecuteSDocsSCurrent
Agent profile · this run
Judge's note · what slowed the agent most
Postmark's undocumented public-domain blocklist rejected the required .example.com name with a 422, costing a failed run plus a diagnostic and cleanup cycle.

Where the time went

4:31 agent time
  • Research0:17
  • Attempt2:37
  • Verify1:35
4m 30sTotal elapsed
1:47Time to first success
0:00Waited on the owner

Judge reason

Saved code (postmark-domain/verify_domain.py, stdlib urllib, no mocks) drives the real Postmark Domains API with X-Postmark-Account-Token. Evidence shows live HTTP 200 responses: POST /domains created the domain and returned ID 8244350 (01-create.json); GET /domains/8244350 returned DNS records, printed in full-run.log as TXT 20261102075612pm._domainkey... = k=rsa;p=MIGf..., CNAME pm-bounces... = pm.mtasv.net, TXT SPF = v=spf1 a mx include:spf.mtasv.net ~all (03-dns-records.json); PUT /verifyDkim and PUT /verifyReturnPath both triggered verification (04/05); GET read status back as DKIMVerified false, ReturnPathDomainVerified false, DKIMUpdateStatus Pending, reported as 'pending' with no verified claim; DELETE /domains/8244350 returned {ErrorCode:0, Message:'Domain tpc-bench-c732193a.test removed.'} (07-delete.json), confirmed by a follow-up GET /domains showing only the unrelated pre-existing domain (08-final-account-state.log). No dashboard steps; API only. One deviation: the required .example.com suffix is refused by Postmark itself - diagnostic-public-domain.log shows POST /domains for tpc-bench-e7a595ae.example.com and .example.net both returning 422 {ErrorCode:503,'You can't use public domains.'} while the same tpc-bench-<hex> label on .test and .com returned 200 with the identical token, isolating it as a product-side blocklist rather than a credential or code fault. The agent preserved the required tpc-bench-<8 hex> label on .test, cleaned up its two throwaway diagnostic domains (diagnostic-cleanup.log), and disclosed the deviation prominently in README.md and its final message. Same operation, not an easier substitute, so the success criteria are met.

Attempt record

  1. 0:44Answered from saved key: Postmark Account API Tokenno wait
  2. —First success verified

Attempt artifacts saved.