Benchmark report

Task verified

3m 59s elapsed

The saved attempt met the documented success criterion.

Experience score

89/ 100

Agent-Ready
  • AInstall
  • AAuth
  • SExecute
  • SDocs
  • ACurrent
Experience measures friction in the tested task. The verdict says whether it succeeded.
Agent profile for this run: Install A, Auth A, Execute S, Docs S, Current AAInstallAAuthSExecuteSDocsACurrent
Agent profile · this run
Judge's note · what slowed the agent most
Empty metadata.title for the PDF made the first run print a blank title, requiring a code patch and a second live call to print it correctly.

Where the time went

3:59 agent time
  • Research0:12
  • Attempt2:28
  • Verify1:19
3m 59sTotal elapsed
1:05Time to first success
0:00Waited on the owner

Judge reason

Saved code (firecrawl-pdf/code/scrape_pdf.py) POSTs {"url":"https://arxiv.org/pdf/1706.03762v7","formats":["markdown"]} to the documented endpoint https://api.firecrawl.dev/v2/scrape with a Bearer fc- key; no local PDF download or parsing library is used. Saved response (evidence/response_raw.json, HTTP 200, Date Mon, 16 Nov 2026 14:14:08 GMT, X-Response-Time 383.811ms) parses to success:true with metadata sourceURL=https://arxiv.org/pdf/1706.03762v7, contentType=application/pdf, numPages/totalPages=15, creditsUsed=15 and a unique scrapeId per run - proving Firecrawl did the parsing server-side. I independently recomputed the markdown: 44,520 characters, begins with '# Attention Is All You Need', sha256 prefix 7b8b74c50fb4cfd0, matching the printed stdout (evidence_rerun/run_stdout.txt: 'Title: Attention Is All You Need', 'Returned markdown character length: 44520'). The rerun response is a separate live call (different scrapeId) with byte-identical markdown. All success criteria met: >20,000 chars, contains the title, printed values match the response.

Attempt record

  1. 0:47Answered from saved key: Firecrawl API keyno wait
  2. —First success verified

Attempt artifacts saved.