The Prompting Company

Benchmark report

Task verified

8m 15s elapsed

The saved attempt met the documented success criterion.

Experience score

83/ 100

Needs Alignment
  • AInstall
  • BAuth
  • SExecute
  • SDocs
  • SCurrent
Experience measures friction in the tested task. The verdict says whether it succeeded.
Agent profile for this run: Install A, Auth B, Execute S, Docs S, Current SAInstallBAuthSExecuteSDocsSCurrent
Agent profile · this run
Judge's note · what slowed the agent most
The benchmark-request for an API key hung to a ~300s timeout — most of the 362s to first success — before the agent used the documented keyless search tier.

Where the time went

5:00 on the owner · 3:15 agent time
Waited on the owner 5:00
  • Starting0:01
  • Research0:11
  • Attempt1:51
  • Verify1:13
8m 15sTotal elapsed
6:02Time to first success
5:00Waited on the owner

Judge reason

Saved search.py POSTs the exact query 'PostgreSQL transaction isolation levels' to Firecrawl's documented web search endpoint (POST https://api.firecrawl.dev/v2/search) via urllib and asserts a nonempty title and absolute http(s) URL. run_output.txt shows HTTP status 200, success: True, result count 5, FIRST RESULT TITLE 'Documentation: 18: 13.2. Transaction Isolation - PostgreSQL', URL https://www.postgresql.org/docs/current/transaction-iso.html; response.json holds the matching documented shape {success, data.web[5], creditsUsed: 2, id 01a13caa-44cf-...} with first entry position 1 identical to the printed values. Liveness is corroborated by raw HTTP evidence: curl_verbatim_response.txt and keyless_probe.txt carry real HTTP/2 200 status lines, server date/etag/x-response-time headers and three distinct Firecrawl request ids (…44cf, …6523, …05fb) across three calls, so nothing is mocked or hard-coded. The agent requested a key via benchmark-request (transcript tool result {"outcome": "timed_out"}) and then used Firecrawl's own keyless free tier, documented in the vendor's auth.md: 'Search, scrape and parse can succeed without a key'; the code still sends Authorization: Bearer when FIRECRAWL_API_KEY is set. No scraping/local-search substitution occurred.

Attempt record

  1. 0:41Asked the owner for Firecrawl API keywaited 300s · no response within the window
  2. —First success verified

Attempt artifacts saved.