Benchmark report

Task verified

3m 57s elapsed

The saved attempt met the documented success criterion.

Experience score

96/ 100

Agent-Ready
  • SInstall
  • SAuth
  • SExecute
  • ADocs
  • SCurrent
Experience measures friction in the tested task. The verdict says whether it succeeded.
Agent profile for this run: Install S, Auth S, Execute S, Docs A, Current SSInstallSAuthSExecuteADocsSCurrent
Agent profile · this run
Judge's note · what slowed the agent most
A 404 on the api-reference Extract page meant piecing the request schema together from the quickstart, llms.txt and advanced-settings markdown.

Where the time went

3:57 agent time
  • Research0:16
  • Attempt2:10
  • Verify1:31
3m 57sTotal elapsed
1:20Time to first success
0:00Waited on the owner

Judge reason

Verified live use of Parallel's GA Extract API. Transcript step 42 runs `./.venv/bin/python -I extract_pdf.py` with the key file from benchmark-request; the SDK call client.extract(urls=["https://arxiv.org/pdf/1706.03762v7"], objective=..., advanced_settings={"full_content":{"max_chars_per_result":200000}}) hit POST https://api.parallel.ai/v1/extract. Saved evidence/extract_response.json is the raw model_dump: extract_id extract_91cb3189336e29444d2afb7f098f2901, session_id session_91cb..., usage [{sku_extract_excerpts:1}], errors [], results[0].url == the arXiv PDF URL, title "Attention Is All You Need", acquired_at 2026-11-11. full_content is 41522 chars (>20,000) and byte-identical to evidence/full_content.md (confirmed independently); it contains the title and genuine paper body through Figure 5 / page 15, so it is not fabricated or hard-coded. Printed values in evidence/run_output.txt (TITLE: Attention Is All You Need, TEXT LENGTH: 41522) match the transcript tool output at step 43 verbatim and match the saved response. No local PDF download or local parser is used anywhere in extract_pdf.py. Capability is GA: docs state /v1/extract is for all new integrations and only /v1beta/extract is legacy, so the beta/preview clause does not apply.

Attempt record

  1. 1:01Answered from saved key: PARALLEL_API_KEYno wait
  2. —First success verified

Attempt artifacts saved.