Benchmark report

Task verified

7m 17s elapsed

The saved attempt met the documented success criterion.

Experience score

86/ 100

Agent-Ready
  • SInstall
  • SAuth
  • AExecute
  • ADocs
  • SCurrent
Experience measures friction in the tested task. The verdict says whether it succeeded.
Agent profile for this run: Install S, Auth S, Execute A, Docs A, Current SSInstallSAuthAExecuteADocsSCurrent
Agent profile · this run
Judge's note · what slowed the agent most
Loops' send API needs a pre-published template, and the create/LMX/expectedRevisionId flow is not in the docs, so the agent reverse-engineered it from app.loops.so/openapi.json.

Where the time went

7:17 agent time
  • Research0:16
  • Attempt5:12
  • Verify1:49
7m 17sTotal elapsed
4:04Time to first success
0:00Waited on the owner

Judge reason

Saved code loops-task/send-transactional.mjs POSTs https://app.loops.so/api/v1/transactional with Authorization: Bearer $LOOPS_API_KEY and body {email:'test@example.com', transactionalId:'cmvh4vwnu00ad0jz8qoion13l', dataVariables:{customerName:'Benchmark Tester', orderNumber:'ORD-20261020-0001'}}; the saved request in evidence/send-response.json matches that code byte-for-byte. Transcript line 144 shows the live run (node send-transactional.mjs, 20:27:02Z) producing HTTP 200 OK with rawBody {"success":true} and genuine live headers (cf-ray a4dac2c91fd1ff09-PDX, x-ratelimit-remaining 8, date Tue, 20 Oct 2026 20:27:02 GMT) - not mocked or hard-coded. Loops is template-based: the subject 'Loops API benchmark: transactional send' and plain-text LMX body were created and published through the API only (POST /v1/transactional-emails 201, POST /v1/email-messages/{id} 200 with emailFormat 'plain', POST /v1/transactional-emails/{id}/publish 200 returning dataVariables [customerName, orderNumber]), all captured in evidence/setup-responses.json; no SMTP and no dashboard. Sender was discovered via the API (GET /v1/transactional-emails -> GET /v1/email-messages/{publishedEmailMessageId} -> fromName 'Prompting Co', fromEmail local part 'rick' on the team's verified domain), and auth was confirmed by GET /v1/api-key -> 200 {"success":true,"teamName":"Prompting Co"}. Printed output matches the saved response: 'HTTP status: 200 OK', body {"success":true}, and an explicit statement that the API returned no message identifier - correct for this endpoint's documented {"success": true} shape. Exactly one send occurred (one POST /v1/transactional in the whole transcript). Recipient note: Loops documents no test recipient; the agent did request one through the benchmark input mechanism (transcript line 40) and received {"outcome":"skipped","message":"No one answers requests during benchmark campaigns..."}, then fell back to test@example.com, the address used in Loops' own openapi.json examples and an RFC 2606 reserved domain with a null MX, so no real mailbox could receive it - the fallback branch was attempted and unavailable, and the no-real-mailbox requirement is met.

Attempt record

  1. 1:23Requested Transactional email recipient addressdeclined automatically: campaigns don't wait for people
  2. —First success verified

Attempt artifacts saved.