Benchmark report

Task verified

13m 45s elapsed

The saved attempt met the documented success criterion.

Experience score

78/ 100

Needs Alignment
  • AInstall
  • SAuth
  • BExecute
  • SDocs
  • SCurrent
Experience measures friction in the tested task. The verdict says whether it succeeded.
Agent profile for this run: Install A, Auth S, Execute B, Docs S, Current SAInstallSAuthBExecuteSDocsSCurrent
Agent profile · this run
Judge's note · what slowed the agent most
Firecrawl's documented mutual exclusivity of includeDomains and excludeDomains, enforced as HTTP 400, made the requested single-request operation impossible.

Where the time went

13:46 agent time
  • Starting1:33
  • Research0:43
  • Attempt6:46
  • Verify4:43
13m 45sTotal elapsed
—Time to first success
0:00Waited on the owner

Judge reason

The agent made live authenticated calls to POST https://api.firecrawl.dev/v2/search with a benchmark-provided fc- bearer key. evidence/response-include-and-exclude.json shows the exact required body {"query":"MVCC vacuum","includeDomains":["postgresql.org"],"excludeDomains":["wiki.postgresql.org"],"limit":10,"sources":[{"type":"web"}]} rejected with HTTP 400 'includeDomains and excludeDomains cannot both be specified'. I independently re-fetched https://docs.firecrawl.dev/api-reference/endpoint/search.md (HTTP 200, 69120 bytes, byte-identical to the saved copy) and confirmed line 75 'includeDomains and excludeDomains are mutually exclusive', plus the OpenAPI 'Cannot be used with excludeDomains/includeDomains' descriptions and docs/features/search line 292. So the API provably cannot express both filters in one request - the success criteria explicitly treat that as a documented gap, and the agent cited it precisely without fabricating. A live include-only control (evidence/response-include-only.json, HTTP 200, creditsUsed 2, request id 01a1c751-...) returned 10 genuine results (8 www.postgresql.org, 2 wiki.postgresql.org), proving credential/endpoint/filter all work and that the exclusion is the only missing piece; the agent did not filter locally or use site: operators, and correctly did not call benchmark-success.

Attempt record

  1. 6:20Answered from saved key: Firecrawl API keyno wait
  2. —First success verified

Attempt artifacts saved.