Benchmark report

Task verified

6m 53s elapsed

The saved attempt met the documented success criterion.

Experience score

93/ 100

Agent-Ready
  • SInstall
  • SAuth
  • SExecute
  • BDocs
  • SCurrent
Experience measures friction in the tested task. The verdict says whether it succeeded.
Agent profile for this run: Install S, Auth S, Execute S, Docs B, Current SSInstallSAuthSExecuteBDocsSCurrent
Agent profile · this run
Judge's note · what slowed the agent most
Clay's REST docs document routine execution but no way to discover a routine id, forcing a long search before pivoting to the Searches endpoint.

Where the time went

6:54 agent time
  • Research0:28
  • Attempt4:11
  • Verify2:14
6m 53sTotal elapsed
2:35Time to first success
0:00Waited on the owner

Judge reason

Verified from saved artifacts and transcript, not prose. reconcile.sh (clay-task/reconcile.sh) runs live calls to https://api.clay.com/public/v0 with the documented `clay-api-key` header: GET /credits/balance (before), POST /search/query-mode (setup, search_0to0ob86bsZZG9WX8BD), POST /search/query-mode/{id}/run with limit 1 = exactly one company enrichment lookup for stripe.com, GET /credits/balance (after). Transcript idx 140 shows the live run output with HTTP 200 for all four, matching out/http_status.json and the saved bodies. out/enrichment_stripe.json holds a real enriched Stripe record (clay_company_id 26519186, industry, size, revenue, funding, linkedin_url) plus an in-band period_quota (used 287869, resets 2026-11-06) - not a mock or hard-coded value. balance_before.json and balance_after.json are byte-identical (278723.9999999997), difference 0, reported in REPORT.md and result.json with the explanation that query-mode search meters against the separate period_quota rather than spendable data credits. I independently confirmed the pricing claim: developers.clay.com llms-full.txt mentions credits only in the /credits/balance section ('Reading balances does not consume credits', line 17) and the saved openapi.json (18 paths) carries no per-lookup price, so 'documentation states no price per lookup' is supported. I also confirmed the openapi path list contains no routine/function listing endpoint, so the agent's documented gap (routine ids only shown as the placeholder function:t_abc123, discovery only via MCP/CLI OAuth) is real and it correctly used the documented Searches operation instead of guessing an endpoint; the query is verbatim from out/query_reference.md ('include_company_identifiers(("stripe.com", "openai.com"))') narrowed to one company. The key was requested via benchmark-request (idx 59, outcome provided), read only inside scripts via `-H @` from /tmp/benchmark-control/... ; keyfile.path stores only the path. My own grep over /workspace/home for bearer/access_token/api-key patterns found nothing, and no token appears in the transcript.

Attempt record

  1. 1:24Answered from saved key: Clay personal API key (clay-api-key header)no wait
  2. —First success verified

Attempt artifacts saved.