Benchmark report

Task verified

8m 59s elapsed

The saved attempt met the documented success criterion.

Experience score

99/ 100

Agent-Ready
  • SInstall
  • SAuth
  • SExecute
  • SDocs
  • ACurrent
Experience measures friction in the tested task. The verdict says whether it succeeded.
Agent profile for this run: Install S, Auth S, Execute S, Docs S, Current ASInstallSAuthSExecuteSDocsACurrent
Agent profile · this run
Judge's note · what slowed the agent most
Heavy background credit consumption on the shared Apollo account (+55/+47 per 26s with no calls) obscured the 1-credit lookup and forced an extra control run.

Where the time went

9:00 agent time
  • Research0:44
  • Attempt6:21
  • Verify1:55
8m 59sTotal elapsed
2:15Time to first success
0:00Waited on the owner

Judge reason

Saved code (apollo-task/code/reconcile_credits.py, plus curl equivalents in commands.sh) performed the documented sequence live: POST /api/v1/usage_stats/credit_usage_stats, GET /api/v1/organizations/enrich?domain=stripe.com, then the stats POST again after a 25s wait. Saved responses 01/02/03 all record http_status 200; 02 contains a real Stripe record (id 5d0a0fbff6512580bf33a120, primary_domain stripe.com, 9400 employees), so no mock. Transcript idx 45-53 shows the actual run output with statuses {200,200,200}. Readings: lead_credit consumed 5927 -> 5978 (left_over 743 -> 692), difference +51, all other credit types delta 0; report.md and the final output print both readings, the difference, and the documented price (1 credit per organization, cited to docs.apollo.io/reference/organization-enrichment and /docs/api-pricing, fetched at transcript idx 16 and 23) and state they do not agree, attributing the gap to concurrent consumption on a shared account rather than delayed update, backed by a zero-enrichment control run (05_control_background_drift.json: deltas 0,+55,+10,+47 over ~104s with no enrichment call). Exactly one enrichment lookup was made. No access token or bearer token exists anywhere: the key was read from a private file, sent only as an x-api-key header, headers_sent saved as 'x-api-key: [redacted]', and a scan of all 8 artifacts found 0 matches for the key value and no access_token/bearer/refresh_token strings; my own grep of /workspace/home confirms no credential material.

Attempt record

  1. 1:34Answered from saved key: Apollo.io API keyno wait
  2. —First success verified

Attempt artifacts saved.