Benchmark report

Task verified

3m 44s elapsed

The saved attempt met the documented success criterion.

Experience score

98/ 100

Agent-Ready
  • AInstall
  • SAuth
  • SExecute
  • SDocs
  • SCurrent
Experience measures friction in the tested task. The verdict says whether it succeeded.
Agent profile for this run: Install A, Auth S, Execute S, Docs S, Current SAInstallSAuthSExecuteSDocsSCurrent
Agent profile · this run
Judge's note · what slowed the agent most
A PEP 668 externally-managed-environment error on the first pip install forced a venv detour; otherwise the run was unobstructed.

Where the time went

3:44 agent time
  • Starting0:01
  • Research0:12
  • Attempt2:27
  • Verify1:05
3m 44sTotal elapsed
1:20Time to first success
0:00Waited on the owner

Judge reason

Saved code parse_pdf.py POSTs {"urls":["https://arxiv.org/pdf/1706.03762v7"],"text":{"maxCharacters":1000000}} to the documented Exa Contents endpoint https://api.exa.ai/contents with x-api-key auth. evidence/response_full.json is a live 200 response (requestId 0a846a981d99d949e5764be4063deffd, matching Cloudflare headers dated Mon, 16 Nov 2026 14:02:05 GMT, costDollars 0.001); results[0].title == 'Attention Is All You Need' and results[0].text is 40819 chars containing that title, verified independently by re-parsing the JSON. evidence/run_output.txt prints exactly that title and 40819, and parsed_text.txt is byte-identical to the response text. Nothing was downloaded or parsed locally (requests only, no PDF library). A second run with maxAgeHours:0 returned statuses source 'crawled' with identical 40819-char text, confirming server-side fetch+parse by Exa rather than a stale artifact.

Attempt record

  1. 0:45Answered from saved key: Exa API key (EXA_API_KEY)no wait
  2. —First success verified

Attempt artifacts saved.