Benchmark report
Task verified
6m 53s elapsed
The saved attempt met the documented success criterion.
93/ 100
Agent-Ready- SInstall
- SAuth
- SExecute
- BDocs
- SCurrent
Clay's REST docs document routine execution but no way to discover a routine id, forcing a long search before pivoting to the Searches endpoint.
Where the time went
6:54 agent time- Research0:28
- Attempt4:11
- Verify2:14
Judge reason
Verified from saved artifacts and transcript, not prose. reconcile.sh (clay-task/reconcile.sh) runs live calls to https://api.clay.com/public/v0 with the documented `clay-api-key` header: GET /credits/balance (before), POST /search/query-mode (setup, search_0to0ob86bsZZG9WX8BD), POST /search/query-mode/{id}/run with limit 1 = exactly one company enrichment lookup for stripe.com, GET /credits/balance (after). Transcript idx 140 shows the live run output with HTTP 200 for all four, matching out/http_status.json and the saved bodies. out/enrichment_stripe.json holds a real enriched Stripe record (clay_company_id 26519186, industry, size, revenue, funding, linkedin_url) plus an in-band period_quota (used 287869, resets 2026-11-06) - not a mock or hard-coded value. balance_before.json and balance_after.json are byte-identical (278723.9999999997), difference 0, reported in REPORT.md and result.json with the explanation that query-mode search meters against the separate period_quota rather than spendable data credits. I independently confirmed the pricing claim: developers.clay.com llms-full.txt mentions credits only in the /credits/balance section ('Reading balances does not consume credits', line 17) and the saved openapi.json (18 paths) carries no per-lookup price, so 'documentation states no price per lookup' is supported. I also confirmed the openapi path list contains no routine/function listing endpoint, so the agent's documented gap (routine ids only shown as the placeholder function:t_abc123, discovery only via MCP/CLI OAuth) is real and it correctly used the documented Searches operation instead of guessing an endpoint; the query is verbatim from out/query_reference.md ('include_company_identifiers(("stripe.com", "openai.com"))') narrowed to one company. The key was requested via benchmark-request (idx 59, outcome provided), read only inside scripts via `-H @` from /tmp/benchmark-control/... ; keyfile.path stores only the path. My own grep over /workspace/home for bearer/access_token/api-key patterns found nothing, and no token appears in the transcript.
Attempt record
- 1:24Answered from saved key: Clay personal API key (clay-api-key header)no wait
- —First success verified
Attempt artifacts saved.