Benchmark report

Task verified

11m 51s elapsed

The saved attempt met the documented success criterion.

Experience score

94/ 100

Agent-Ready
  • AInstall
  • SAuth
  • SExecute
  • ADocs
  • SCurrent
Experience measures friction in the tested task. The verdict says whether it succeeded.
Agent profile for this run: Install A, Auth S, Execute S, Docs A, Current SAInstallSAuthSExecuteADocsSCurrent
Agent profile · this run
Judge's note · what slowed the agent most
First run at the default-ish limit 10 yielded only ~2 German results, so the agent re-ran at limit 30; checking the docs' 'languages' claim meant downloading a 16MB locations file.

Where the time went

11:52 agent time
  • Starting1:32
  • Research0:15
  • Attempt6:36
  • Verify3:29
11m 51sTotal elapsed
2:13Time to first success
0:00Waited on the owner

Judge reason

Saved code firecrawl-task/search_de.py POSTs https://api.firecrawl.dev/v2/search with body {"query":"Datenbank Transaktionsisolation","country":"DE","location":"Germany","sources":["web"],"limit":30}; evidence/search-output.txt shows 'HTTP 200 success=True' and evidence/search-response.json is a genuine live payload (creditsUsed 6, request id 01a1c750-591b-76dc-95bd-92eef5c9379d, 30 web results with url/title/description/position). I independently re-fetched https://docs.firecrawl.dev/api-reference/endpoint/search.md (identical 69120 bytes to the saved copy) and confirmed the /v2/search request schema documents only 'location' and 'country' geo-targeting; the sole 'languages' array lives under scrapeOptions.location (browser Accept-Language), so no search language parameter exists — the agent sent both documented targeting parameters and explicitly named the missing language parameter, which the success criteria allow. Far more than 3 results are German by title/excerpt (e.g. #2 de.wikipedia 'In der Informatik bezeichnet der Begriff Isolation...', #4 learn.microsoft.com/de-de 'Die Standardisolationsstufe ist READ UNCOMMITTED', #11 datasunrise.com/de, #13 datenbanken-verstehen.de, #16 wiki.floriancode.de, #24 db-berater.de). All 30 results are printed verbatim in API order with no local translation or filtering (the German tally in german-result-count.txt is post-hoc reporting only); 1 result (#10) lacked a description, which the code prints as '(none returned by API)'. Credentials came from benchmark-request (outcome 'provided') and the key was never written into artifacts (leak check showed only fc-YOUR-API-KEY placeholders). Capability is GA, not beta/gated.

Attempt record

  1. 4:43Answered from saved key: Firecrawl API keyno wait
  2. —First success verified

Attempt artifacts saved.