Benchmark report
Task verified
9m 54s elapsed
The saved attempt met the documented success criterion.
81/ 100
Needs Alignment- SInstall
- SAuth
- AExecute
- BDocs
- ACurrent
Hunting for machine-readable docs: thin richapi.ai/api page, several dead OpenAPI URLs, and a stale linkedin_company_search slug to reconcile against the live company_search.
Where the time went
9:54 agent time- Research0:22
- Attempt6:17
- Verify3:15
Judge reason
Verified independently from saved artifacts. code/company_search_germany.py POSTs https://api.richapi.ai/api/v1/company_search with an x-api-key header read from a private file, body {"linkedin_industry_id":"4","geo_id":"101282230","company_size":"201-500","page":1}; stdout shows HTTP 200 and the saved response matches the documented shape in docs/public_openapi.json (/company_search: pagination, elements, error, richapi_meta; live richapi_meta has request_id req-3dfd97c1..., response_time_ms, credits charged 2.5 debiting the wallet 5014.43->5004.43 across runs), so the response is live, not mocked. All three filters are server-side and documented: control probes give totalResultCount 544 (all filters) vs 26,312 without company_size and 13,512 without geo/location, with a different head of results, so nothing was post-filtered. geo_id 101282230 came from the documented POST /geo_id_search {"search":"Germany"} (HTTP 200, {"geoId":"101282230","title":"Germany"}). All 50 returned elements report industry "Software Development"; the printed table (outputs/company_search_germany_page1.stdout.txt) matches the saved JSON exactly (names, location.linkedinText, totalResultCount 544). Two task items cannot be met by the product, and the agent disclosed both with doc citations instead of guessing or post-filtering: /company_search documents no limit/page-size parameter (server pageSize is fixed at 50), so it printed the first 10 of page 1 and saved the full page; and the spec documents only a generic location filter ("Filter companies by location" / geo_id "overrides location") with no headquarters parameter, so 8 of 10 rows show a non-German HQ string (only Parloa and Gorillas show Berlin) -- the report states this gap and cites https://app.richapi.ai/api/openapi and llms.txt. The response schema itself carries no domain/website or employee-count fields (confirmed in the OpenAPI example), and the agent printed "(not in response)" rather than inventing values. No credential leaked: a grep of code/, outputs/ and report.md finds only the header name "x-api-key" and licdn logo URL tokens; the agent's own key-substring scan over the home directory returned no hits, and the only access_token/bearer strings anywhere are field names inside the fetched docs/openapi.json schema.
Attempt record
- 2:05Answered from saved key: RichAPI API keyno wait
- —First success verified
Attempt artifacts saved.