Benchmark report

Task verified

9m 54s elapsed

The saved attempt met the documented success criterion.

Experience score

81/ 100

Needs Alignment
  • SInstall
  • SAuth
  • AExecute
  • BDocs
  • ACurrent
Experience measures friction in the tested task. The verdict says whether it succeeded.
Agent profile for this run: Install S, Auth S, Execute A, Docs B, Current ASInstallSAuthAExecuteBDocsACurrent
Agent profile · this run
Judge's note · what slowed the agent most
Hunting for machine-readable docs: thin richapi.ai/api page, several dead OpenAPI URLs, and a stale linkedin_company_search slug to reconcile against the live company_search.

Where the time went

9:54 agent time
  • Research0:22
  • Attempt6:17
  • Verify3:15
9m 54sTotal elapsed
3:51Time to first success
0:00Waited on the owner

Judge reason

Verified independently from saved artifacts. code/company_search_germany.py POSTs https://api.richapi.ai/api/v1/company_search with an x-api-key header read from a private file, body {"linkedin_industry_id":"4","geo_id":"101282230","company_size":"201-500","page":1}; stdout shows HTTP 200 and the saved response matches the documented shape in docs/public_openapi.json (/company_search: pagination, elements, error, richapi_meta; live richapi_meta has request_id req-3dfd97c1..., response_time_ms, credits charged 2.5 debiting the wallet 5014.43->5004.43 across runs), so the response is live, not mocked. All three filters are server-side and documented: control probes give totalResultCount 544 (all filters) vs 26,312 without company_size and 13,512 without geo/location, with a different head of results, so nothing was post-filtered. geo_id 101282230 came from the documented POST /geo_id_search {"search":"Germany"} (HTTP 200, {"geoId":"101282230","title":"Germany"}). All 50 returned elements report industry "Software Development"; the printed table (outputs/company_search_germany_page1.stdout.txt) matches the saved JSON exactly (names, location.linkedinText, totalResultCount 544). Two task items cannot be met by the product, and the agent disclosed both with doc citations instead of guessing or post-filtering: /company_search documents no limit/page-size parameter (server pageSize is fixed at 50), so it printed the first 10 of page 1 and saved the full page; and the spec documents only a generic location filter ("Filter companies by location" / geo_id "overrides location") with no headquarters parameter, so 8 of 10 rows show a non-German HQ string (only Parloa and Gorillas show Berlin) -- the report states this gap and cites https://app.richapi.ai/api/openapi and llms.txt. The response schema itself carries no domain/website or employee-count fields (confirmed in the OpenAPI example), and the agent printed "(not in response)" rather than inventing values. No credential leaked: a grep of code/, outputs/ and report.md finds only the header name "x-api-key" and licdn logo URL tokens; the agent's own key-substring scan over the home directory returned no hits, and the only access_token/bearer strings anywhere are field names inside the fetched docs/openapi.json schema.

Attempt record

  1. 2:05Answered from saved key: RichAPI API keyno wait
  2. —First success verified

Attempt artifacts saved.