Benchmark report
Task verified
3m 57s elapsed
The saved attempt met the documented success criterion.
96/ 100
Agent-Ready- SInstall
- SAuth
- SExecute
- ADocs
- SCurrent
A 404 on the api-reference Extract page meant piecing the request schema together from the quickstart, llms.txt and advanced-settings markdown.
Where the time went
3:57 agent time- Research0:16
- Attempt2:10
- Verify1:31
Judge reason
Verified live use of Parallel's GA Extract API. Transcript step 42 runs `./.venv/bin/python -I extract_pdf.py` with the key file from benchmark-request; the SDK call client.extract(urls=["https://arxiv.org/pdf/1706.03762v7"], objective=..., advanced_settings={"full_content":{"max_chars_per_result":200000}}) hit POST https://api.parallel.ai/v1/extract. Saved evidence/extract_response.json is the raw model_dump: extract_id extract_91cb3189336e29444d2afb7f098f2901, session_id session_91cb..., usage [{sku_extract_excerpts:1}], errors [], results[0].url == the arXiv PDF URL, title "Attention Is All You Need", acquired_at 2026-11-11. full_content is 41522 chars (>20,000) and byte-identical to evidence/full_content.md (confirmed independently); it contains the title and genuine paper body through Figure 5 / page 15, so it is not fabricated or hard-coded. Printed values in evidence/run_output.txt (TITLE: Attention Is All You Need, TEXT LENGTH: 41522) match the transcript tool output at step 43 verbatim and match the saved response. No local PDF download or local parser is used anywhere in extract_pdf.py. Capability is GA: docs state /v1/extract is for all new integrations and only /v1beta/extract is legacy, so the beta/preview clause does not apply.
Attempt record
- 1:01Answered from saved key: PARALLEL_API_KEYno wait
- —First success verified
Attempt artifacts saved.