Benchmark report

Task verified

5m 33s elapsed

The saved attempt met the documented success criterion.

Experience score

96/ 100

Agent-Ready
  • SInstall
  • SAuth
  • SExecute
  • ADocs
  • SCurrent
Experience measures friction in the tested task. The verdict says whether it succeeded.
Agent profile for this run: Install S, Auth S, Execute S, Docs A, Current SSInstallSAuthSExecuteADocsSCurrent
Agent profile · this run
Judge's note · what slowed the agent most
Grepping the 173KB query-reference text to confirm company_size/industry/locations field names, plus one ModuleNotFoundError from a flat file layout.

Where the time went

5:34 agent time
  • Research0:25
  • Attempt3:16
  • Verify1:51
5m 33sTotal elapsed
2:06Time to first success
0:00Waited on the owner

Judge reason

Saved code (clay-task/code/clay_client.py + company_search.py) makes real authenticated calls to Clay's documented Public API at https://api.clay.com/public/v0 with the clay-api-key header read from a private key file. Transcript idx 37 shows GET /me -> 200 and GET /search/query-mode/reference -> 200; idx 66 shows POST /search/query-mode -> HTTP 200 (search_id search_0to0lvpqZtURw4fjDRD) and POST /search/query-mode/{id}/run with body {'limit':10} -> HTTP 200. Filtering is server-side in the created query: 'select from companies where industry in ("Software Development") and locations.any(is_headquarters = true and country_name = "Germany") and company_size in ("201-500")' — no post-filtering (code prints rows as returned). The field names and enum values are confirmed by the live reference response saved at responses/reference.json (company_size enum includes "201-500"; the reference even shows the exact industry/locations.any pattern). responses/03_run_page1.json contains data[10], has_more true, source_type companies, period_quota; all 10 rows have industry='Software Development', country='Germany', size='201-500', and the printed outputs_run.txt values match the saved JSON exactly (independently re-parsed). No total/count field exists in the response, and the report says so. evidence_run_page1.json is byte-identical to responses/03_run_page1.json. A key-value scan over all artifacts (transcript idx 90-91) found no occurrence of the key; my own grep over /workspace/home found only the literal words 'access_token'/'clay-api-key' in code identifiers and $CLAY_KEY_FILE references, never a credential value. No mocks or hard-coded data.

Attempt record

  1. 1:12Answered from saved key: Clay Public API keyno wait
  2. —First success verified

Attempt artifacts saved.