The Prompting Company

Benchmark report

Task verified

8m 33s elapsed

The saved attempt met the documented success criterion.

Experience score

79/ 100

Needs Alignment
  • SInstall
  • BAuth
  • SExecute
  • ADocs
  • BCurrent
Experience measures friction in the tested task. The verdict says whether it succeeded.
Agent profile for this run: Install S, Auth B, Execute S, Docs A, Current BSInstallBAuthSExecuteADocsBCurrent
Agent profile · this run
Judge's note · what slowed the agent most
The benchmark API-key request hung for ~5 of the ~6.4 minutes (22:35:48 to 22:40:49) before timing out, then the SDK rejected keyless construction and required a rewrite.

Where the time went

5:00 on the owner · 3:34 agent time
Waited on the owner 5:00
  • Starting0:01
  • Research0:11
  • Attempt2:07
  • Verify1:15
8m 33sTotal elapsed
6:22Time to first success
5:00Waited on the owner

Judge reason

Saved code project/search_firecrawl.py POSTs {"query":"PostgreSQL transaction isolation levels","limit":5,"sources":["web"]} to the documented https://api.firecrawl.dev/v2/search. evidence/response.json records http_status 200, success:true, creditsUsed/id, and 5 results in the documented {data:{web:[{url,title,description,position}]}} shape; an independent curl (curl_status.txt HTTP:200, curl_headers.txt HTTP/2 200, curl_response.json) returns the same 5 results. stdout.txt prints the first result title 'Documentation: 18: 13.2. Transaction Isolation - PostgreSQL' and absolute URL https://www.postgresql.org/docs/current/transaction-iso.html, exactly matching web[0] in both response files. No mocking or hard-coding: the script parses the live HTTP response and exits nonzero on empty/bad results. The benchmark credential request was made (transcript idx 31) and returned {"outcome":"timed_out"}; the agent then used the keyless path explicitly documented in docs/features-search.md ('No API key needed to get started - add one for higher rate limits'), which the goal permits for a documented keyless task.

Attempt record

  1. 0:49Asked the owner for Firecrawl API keywaited 300s · no response within the window
  2. —First success verified

Attempt artifacts saved.