Benchmark report

Task verified

4m 35s elapsed

The saved attempt met the documented success criterion.

Experience score

95/ 100

Agent-Ready
  • SInstall
  • SAuth
  • SExecute
  • ADocs
  • ACurrent
Experience measures friction in the tested task. The verdict says whether it succeeded.
Agent profile for this run: Install S, Auth S, Execute S, Docs A, Current ASInstallSAuthSExecuteADocsACurrent
Agent profile · this run
Judge's note · what slowed the agent most
The undocumented rejection of example.com ('You can't use public domains', 422/503) forced a failed run, a base-domain probe, and a patch to fall back to .invalid.

Where the time went

4:35 agent time
  • Research0:19
  • Attempt2:30
  • Verify1:46
4m 35sTotal elapsed
1:48Time to first success
0:00Waited on the owner

Judge reason

Saved runnable code (postmark_domain_task.py, stdlib urllib against https://api.postmarkapp.com, token read from a private file) plus evidence/run_output.txt and evidence/api_transcript.json show a live, unmocked lifecycle: POST /domains {"Name":"tpc-bench-97766be6.example.com"} -> HTTP 422 {"ErrorCode":503,"Message":"You can't use public domains."}, then POST /domains -> HTTP 200 ID=8274831 Name=tpc-bench-97766be6.invalid; GET /domains/8274831 -> 200 listing every returned DNS record (TXT 20261109202601pm._domainkey... DKIM pending key; CNAME pm-bounces.tpc-bench-97766be6.invalid -> pm.mtasv.net; TXT SPF v=spf1 a mx include:spf.mtasv.net ~all); PUT /verifyDkim and PUT /verifyReturnPath -> 200; GET read-back -> DKIMVerified false, ReturnPathDomainVerified false, DKIMUpdateStatus Pending, reported as 'pending (not yet verified)' with no verified claim; DELETE /domains/8274831 -> 200 {"ErrorCode":0,"Message":"Domain tpc-bench-97766be6.invalid removed."} confirmed by GET -> 422 {"ErrorCode":510,"This domain was not found."}. Timestamps in api_transcript.json span 20:26:01.02-20:26:01.99 with realistic per-call latencies; no dashboard or SMTP was used. The only deviation from the literal spec is the TLD: evidence/base_domain_probe.txt shows live POSTs for example.com/.org/.net all rejected 422/503 while .invalid and .test succeeded, so the exact name is impossible on this product; the agent preserved the exact tpc-bench-97766be6 label and used a TLD for which DNS likewise cannot be published, satisfying every success criterion (tpc-bench domain created with identifier, DNS records retrieved and listed, verification operations called, status read back as pending, deletion response shown).

Attempt record

  1. 0:32Answered from saved key: Postmark Account API Token (X-Postmark-Account-Token)no wait
  2. —First success verified

Attempt artifacts saved.