The Prompting Company

Benchmark report

Task verified

8m 39s elapsed

The saved attempt met the documented success criterion.

Experience score

79/ 100

Needs Alignment
  • SInstall
  • BAuth
  • SExecute
  • ADocs
  • BCurrent
Experience measures friction in the tested task. The verdict says whether it succeeded.
Agent profile for this run: Install S, Auth B, Execute S, Docs A, Current BSInstallBAuthSExecuteADocsBCurrent
Agent profile · this run
Judge's note · what slowed the agent most
The benchmark credential request hung ~5 minutes before timing out, and the docs' keyless SDK example failed, forcing a switch to raw HTTP.

Where the time went

5:00 on the owner · 3:40 agent time
Waited on the owner 5:00
  • Research0:11
  • Attempt2:16
  • Verify1:12
8m 39sTotal elapsed
6:11Time to first success
5:00Waited on the owner

Judge reason

Saved search.py POSTs the exact query "PostgreSQL transaction isolation levels" to the documented endpoint https://api.firecrawl.dev/v2/search. evidence/run_output.txt shows HTTP 200, success: True, result count 5, first result title 'Documentation: 18: 13.2. Transaction Isolation - PostgreSQL' / https://www.postgresql.org/docs/current/transaction-iso.html; evidence/response.json holds the full documented shape (success, data.web[] with url/title/description/position, creditsUsed 2, server id) matching the printed first result, and evidence/http_meta.json records status 200 with live server headers (Date Wed, 14 Oct 2026 23:29:40 GMT, X-Response-Time 425ms). An independent cURL call (evidence/curl_raw_response.txt) returned HTTP/2 200 with a different etag/timestamp (23:29:52, 360ms) and the same first result, so the response is live, not mocked or hard-coded. The API key requested via benchmark-request timed out, but Firecrawl's own docs fetched in-transcript document the keyless path ('No API key needed to get started - add one for higher rate limits'; auth.md: 'Search, scrape and parse can succeed without a key'), so the keyless live call satisfies the documented operation. Setup also completed cleanly: venv + firecrawl-py 4.50.0 installed; the SDK variant could not run because firecrawl-py 4.50.0 raises 'No API key provided' at construction.

Attempt record

  1. 0:57Asked the owner for Firecrawl API keywaited 300s · no response within the window
  2. —First success verified

Attempt artifacts saved.