Benchmark report

Task verified

6m 43s elapsed

The saved attempt met the documented success criterion.

Experience score

99/ 100

Agent-Ready
  • SInstall
  • SAuth
  • SExecute
  • SDocs
  • ACurrent
Experience measures friction in the tested task. The verdict says whether it succeeded.
Agent profile for this run: Install S, Auth S, Execute S, Docs S, Current ASInstallSAuthSExecuteSDocsACurrent
Agent profile · this run
Judge's note · what slowed the agent most
Undocumented null title field for direct PDF downloads: the first run printed the Google attribution line as the title, needing a heuristic fix and a second live call.

Where the time went

6:43 agent time
  • Starting0:25
  • Research0:46
  • Attempt4:01
  • Verify1:30
6m 43sTotal elapsed
2:01Time to first success
0:00Waited on the owner

Judge reason

Saved code fetch_pdf.py POSTs {"urls":["https://arxiv.org/pdf/1706.03762v7"],"format":"markdown","ttl":0} to the documented endpoint https://api.fetch.tinyfish.ai with the X-API-Key header; only stdlib json/os/sys/urllib are imported, so no local PDF library parsed anything. run_output.txt records HTTP 200, latency_ms 11104.3, TITLE 'Attention Is All You Need', TEXT LENGTH 39613 and contains-title True. I independently parsed evidence/response.json: results[0].text is 39613 chars (>20,000), contains 'Attention Is All You Need', final_url is the target PDF, errors is empty, and evidence/paper.md matches the response text exactly (39613 chars). Content is genuine server-side PDF extraction - it includes 'Scaled Dot-Product Attention', 'Multi-Head Attention', 'Positional Encoding', 'BLEU' and ends at the page-15 Figure 5 caption with characteristic one-word-per-line attention-figure artifacts. Two live runs (response_run1.json vs response.json) returned byte-identical text with different latencies (6239.7 vs 11104.3 ms), confirming live calls rather than a hard-coded body. Docs fetched in-transcript list PDF under supported content types with only the unused 'highlights' parameter marked Beta, so no beta-gated capability was used.

Attempt record

  1. 2:54Answered from saved key: TinyFish API key (TINYFISH_API_KEY)no wait
  2. —First success verified

Attempt artifacts saved.