Benchmark report

Task verified

3m 37s elapsed

The saved attempt met the documented success criterion.

Experience score

99/ 100

Agent-Ready
  • SInstall
  • SAuth
  • SExecute
  • SDocs
  • ACurrent
Experience measures friction in the tested task. The verdict says whether it succeeded.
Agent profile for this run: Install S, Auth S, Execute S, Docs S, Current ASInstallSAuthSExecuteSDocsACurrent
Agent profile · this run
Judge's note · what slowed the agent most
The documented results[0].title field came back null for the PDF, forcing a re-run after patching the script to pull the title from the extracted markdown.

Where the time went

3:38 agent time
  • Starting0:01
  • Research0:26
  • Attempt1:48
  • Verify1:23
3m 37sTotal elapsed
0:56Time to first success
0:00Waited on the owner

Judge reason

Saved evidence proves a live product-side parse. tinyfish-pdf/parse_pdf.py POSTs {"urls":["https://arxiv.org/pdf/1706.03762v7"],"format":"markdown","ttl":0} to the documented endpoint https://api.fetch.tinyfish.ai with an X-API-Key header (key read from the benchmark-provided private file, never embedded). response.json records HTTP 200 with live-looking server headers (Date Mon, 16 Nov 2026 14:22:52 GMT, Server uvicorn, X-Request-Id, X-Ratelimit-Remaining 998, CloudFront Miss), errors: [], and results[0] with url/final_url = the arXiv PDF, format markdown, latency_ms 2128, and text of 39,613 characters containing 'Attention Is All You Need' plus full paper body (28 'transformer' hits, attention-visualization figures, ending at page 15). extracted.md byte-matches that text; run_output.txt prints TITLE (document): Attention Is All You Need and TEXT LENGTH: 39613, matching the response exactly. No PDF download or local parser appears anywhere in the transcript (no pypdf/pdfplumber/pdftotext/curl of arxiv); only the URL was sent. Docs fetched in-transcript show PDF text extraction is GA and mark only the unused 'highlights' field beta, so nothing gates the capability.

Attempt record

  1. 1:00Answered from saved key: TinyFish API key (TINYFISH_API_KEY)no wait
  2. —First success verified

Attempt artifacts saved.