Benchmark report

Task verified

3m 59s elapsed

The saved attempt met the documented success criterion.

Experience score

98/ 100

Agent-Ready
  • AInstall
  • SAuth
  • SExecute
  • SDocs
  • SCurrent
Experience measures friction in the tested task. The verdict says whether it succeeded.
Agent profile for this run: Install A, Auth S, Execute S, Docs S, Current SAInstallSAuthSExecuteSDocsSCurrent
Agent profile · this run
Judge's note · what slowed the agent most
Only real friction was PEP 668 blocking a system-wide pip install, costing one extra command to create a venv; Exa itself required no workarounds.

Where the time went

3:59 agent time
  • Research0:12
  • Attempt2:14
  • Verify1:32
3m 59sTotal elapsed
1:10Time to first success
0:00Waited on the owner

Judge reason

Saved code posts the exact target URL to Exa's documented GA endpoint: raw_rest_call.py and parse_pdf.py both send {"urls":["https://arxiv.org/pdf/1706.03762v7"],"text":true,...} to POST https://api.exa.ai/contents with the x-api-key header. Evidence proves a live server-side parse, not a mock: raw_rest_response_meta.json records HTTP 200 with genuine Exa/Cloudflare headers (X-Request-Id 3bea65067e5a721e8fb7d394e9ae2030, X-Exa-Region us-west-2, Cf-Ray a5b72bd0bf965ed4-PDX, Date Mon 16 Nov 2026 14:26:31 GMT). I re-parsed both saved bodies myself: each result[0] has url=https://arxiv.org/pdf/1706.03762v7, title='Attention Is All You Need', text length 40819 (>20,000), title present in text, and real paper content (Multi-Head Attention, scaled dot-product, BLEU, softmax). Printed stdout (run_output.txt, raw_rest_run_output.txt) matches exactly: title 'Attention Is All You Need', 40819 chars. Parsing was done by Exa — statuses show source='crawled' on the first call then 'cached' on the replay, costDollars 0.001 billed for contents.text, and only exa-py was installed (no pypdf/pdfminer anywhere in the transcript; the PDF was never downloaded locally).

Attempt record

  1. 0:44Answered from saved key: EXA_API_KEYno wait
  2. —First success verified

Attempt artifacts saved.