Benchmark report

Task verified

16m 36s elapsed

The saved attempt met the documented success criterion.

Experience score

96/ 100

Agent-Ready
  • AInstall
  • SAuth
  • SExecute
  • SDocs
  • ACurrent
Experience measures friction in the tested task. The verdict says whether it succeeded.
Agent profile for this run: Install A, Auth S, Execute S, Docs S, Current AAInstallSAuthSExecuteSDocsACurrent
Agent profile · this run
Judge's note · what slowed the agent most
Waiting on the SSE run: docs predicted 5-20 minutes but it finished in 46s, and sleep was blocked, so the agent burned ~13 idle polling turns plus a redundant Monitor.

Where the time went

16:37 agent time
  • Starting1:39
  • Research0:51
  • Attempt12:30
  • Verify1:37
16m 36sTotal elapsed
2:30Time to first success
0:00Waited on the owner

Judge reason

Saved evidence proves a live authenticated call to TinyFish's documented answer operation. commands.txt and research_cited_answer.py show POST https://agent.tinyfish.ai/v1/automation/run-research with X-API-Key and body {"query":"What is PostgreSQL's default transaction isolation level?","mode":"deep"} - the exact endpoint, auth and request shape in the fetched official docs (research-api.md, reference.md). evidence/raw_sse.log holds a genuine 33KB SSE trace (HTTP 200, X-Research-Run-Id 01a1c767-7f9b-70c8-b8fa-cf436cfa6345, created/run_status/plan_updated/sources_searching/synthesis_started/final_result/done events with server stream ids timestamped 2026-11-10T21:40Z). I re-parsed that raw stream and its final_result event is byte-identical to evidence/final_result.json in both 'result' and 'citations'. The product-generated report states "PostgreSQL's default transaction isolation level is **Read Committed**" and the response's citations array holds two absolute HTTPS URLs (https://www.postgresql.org/docs/current/transaction-iso.html, https://postgresqlco.nf/doc/en/param/transaction_isolation/). evidence/run_stdout.log prints that same answer and both citation URLs, matching the saved response; run_stderr.log is empty. No mocking or hard-coded data: the answer text, subquestion plan and citations all arrive from the server stream. Docs mark the Research API as generally available (the 403 'Research must be enabled' line is an error condition, not a beta/preview/early-access marking) and the key returned 200 with no access request.

Attempt record

  1. 3:58Answered from saved key: TINYFISH_API_KEYno wait
  2. —First success verified

Attempt artifacts saved.