Benchmark report

Task verified

9m 30s elapsed

The saved attempt met the documented success criterion.

Experience score

96/ 100

Agent-Ready
  • AInstall
  • SAuth
  • SExecute
  • SDocs
  • ACurrent
Experience measures friction in the tested task. The verdict says whether it succeeded.
Agent profile for this run: Install A, Auth S, Execute S, Docs S, Current AAInstallSAuthSExecuteSDocsACurrent
Agent profile · this run
Judge's note · what slowed the agent most
Exa returning an empty results[0].title for the PDF, forcing the agent to derive the title from the parsed text and re-run the request.

Where the time went

9:30 agent time
  • Starting0:01
  • Research0:40
  • Attempt4:12
  • Verify4:37
9m 30sTotal elapsed
1:00Time to first success
0:00Waited on the owner

Judge reason

Saved code (exa-pdf-parse/parse_pdf.py) POSTs {"urls":["https://arxiv.org/pdf/1706.03762v7"],"text":true,"livecrawlTimeout":60000} to Exa's documented Contents API (https://api.exa.ai/contents) with x-api-key auth, using only stdlib urllib — no local PDF library. Saved run_output.txt shows HTTP 200, statuses success, Title 'Attention Is All You Need', Length 39463 chars. I independently re-parsed evidence/contents_response.json: requestId e34ff8dc48b7b3412494031edea11965, costDollars $0.001, results[0].url == the requested arXiv PDF, text length exactly 39463 (>20,000) containing 'Attention Is All You Need' plus genuine body text through page 15 ('Scaled Dot-Product Attention', Figure 5 caption) — printed title and length match the response. Transcript shows the key was obtained via benchmark-request and the live call made at index 20/36; nothing mocked or hard-coded. Exa returned an empty results[0].title, so the script derives the title at runtime from the first non-notice line of the Exa-parsed text and prints its source — disclosed, not fabricated.

Attempt record

  1. 1:30Answered from saved key: Exa API keyno wait
  2. —First success verified

Attempt artifacts saved.