Benchmark report

Task verified

9m 9s elapsed

The saved attempt met the documented success criterion.

Experience score

100/ 100

Agent-Ready
  • SInstall
  • SAuth
  • SExecute
  • SDocs
  • SCurrent
Experience measures friction in the tested task. The verdict says whether it succeeded.
Agent profile for this run: Install S, Auth S, Execute S, Docs S, Current SSInstallSAuthSExecuteSDocsSCurrent
Agent profile · this run
Judge's note · what slowed the agent most
Essentially nothing: docs fetch, key request, write script, one live call — benchmark-first-success recorded at 44.6s with no retries or errors.

Where the time went

9:10 agent time
  • Starting0:29
  • Research0:55
  • Attempt5:27
  • Verify2:18
9m 9sTotal elapsed
0:45Time to first success
0:00Waited on the owner

Judge reason

Saved code pdf-parse/src/parse_pdf.py POSTs {"url":"https://arxiv.org/pdf/1706.03762v7","formats":["markdown"],"parsers":["pdf"]} to Firecrawl's documented POST https://api.firecrawl.dev/v2/scrape with a bearer key from a private file; it never downloads or locally parses the PDF. evidence/run-output.txt shows 'HTTP 200 ... success: True, sourceURL: https://arxiv.org/pdf/1706.03762v7, contentType: application/pdf, numPages: 15, Title: Attention Is All You Need, Character length of returned markdown: 44677'. I independently re-parsed evidence/scrape-response.json (46546 bytes, sha256 9c218aa...): data.markdown is 44677 chars (>20000), begins '# Attention Is All You Need', and metadata carries server-side markers scrapeId 01a1c757-269f-742f-b60d-33fbfaca7f75, indexId, cacheState 'miss', creditsUsed 15 — a real authenticated call, not a mock. Content spot-checks at offsets 20k and the tail match the genuine paper with Firecrawl-specific PDF-to-markdown artifacts (LaTeX $$ blocks, attention figures rendered as tables), and evidence/attention-is-all-you-need.md is byte-identical to the response markdown. Printed title and length match the response exactly.

Attempt record

  1. 5:18Answered from saved key: Firecrawl API keyno wait
  2. —First success verified

Attempt artifacts saved.