Benchmark report

Task verified

11m 30s elapsed

The saved attempt met the documented success criterion.

Experience score

94/ 100

Agent-Ready
  • AInstall
  • SAuth
  • SExecute
  • ADocs
  • SCurrent
Experience measures friction in the tested task. The verdict says whether it succeeded.
Agent profile for this run: Install A, Auth S, Execute S, Docs A, Current SAInstallSAuthSExecuteADocsSCurrent
Agent profile · this run
Judge's note · what slowed the agent most
Waiting on TinyFish run latency: the last of the five batch runs took 371s to reach COMPLETED, dominating the attempt's wall time.

Where the time went

11:30 agent time
  • Research0:23
  • Attempt9:29
  • Verify1:38
11m 30sTotal elapsed
8:08Time to first success
0:00Waited on the owner

Judge reason

Saved code tinyfish-batch/batch_scrape.py makes exactly one POST https://agent.tinyfish.ai/v1/automation/run-batch with a runs array of all five PostgreSQL 17 URLs; evidence/batch_request.json holds that single body and evidence/batch_response.json the raw live reply {"run_ids":[5 uuids],"error":null} (HTTP 200 logged in run_log.txt). The script then polls GET /v1/runs/{id} in code every 10s until terminal; run_log shows the five runs reaching COMPLETED between 64s and 371s. evidence/runs_full_response.json contains five full run objects whose per-run goal strings map each run_id to its submitted URL (confirming the request-order mapping), all status COMPLETED, created_at identical 2026-11-16T13:16:38Z, with live artifacts (streaming_url, step screenshots, started/finished timestamps). Recomputing len(json.dumps(result)) from the saved runs reproduces the printed lengths exactly (2036, 23995, 16851, 6600, 7842), and the text is genuine PostgreSQL 17 content ('13.2. Transaction Isolation', '3.5. Window Functions'). No single-page submission loop and no synchronous /v1/automation/run call appear anywhere. Per the product's documented shape (and the research plan's own note) run-batch returns a run_ids list rather than a wrapper job id; the agent recorded the returned id set verbatim as the job id and disclosed this, which is product-shape drift, not fabrication. No mocking: auth was proven live by a 404 NOT_FOUND probe with the benchmark-provided key before the real run.

Attempt record

  1. 1:12Answered from saved key: TinyFish API keyno wait
  2. —First success verified

Attempt artifacts saved.