Benchmark report

Task verified

3m 54s elapsed

The saved attempt met the documented success criterion.

Experience score

96/ 100

Agent-Ready
  • SInstall
  • SAuth
  • SExecute
  • ADocs
  • SCurrent
Experience measures friction in the tested task. The verdict says whether it succeeded.
Agent profile for this run: Install S, Auth S, Execute S, Docs A, Current SSInstallSAuthSExecuteADocsSCurrent
Agent profile · this run
Judge's note · what slowed the agent most
Missing advanced_settings schema: the API-reference page and openapi.json 404'd, so the agent read parallel-web source to learn full_content takes a bool.

Where the time went

3:54 agent time
  • Research0:16
  • Attempt2:10
  • Verify1:28
3m 54sTotal elapsed
1:27Time to first success
0:00Waited on the owner

Judge reason

Live call verified: code/extract_pdf.py posts urls=["https://arxiv.org/pdf/1706.03762v7"] with advanced_settings {full_content: true} to Parallel's documented POST https://api.parallel.ai/v1/extract via parallel-web 1.3.7 (x-api-key from a benchmark-provided key file). Transcript idx 45-46 shows the run and its stdout; evidence/extract_response.json holds a genuine response (extract_id extract_6000a3ad765f53d664fc253794437285, session_id, usage sku_extract_excerpts, acquired_at 2026-11-11, errors []). I independently re-parsed it: results[0].url equals the requested PDF URL, title is "Attention Is All You Need", full_content is 41522 chars (>20000) and contains the title; the text is authentic paper body (e.g. the d_model=512 / N=6 decoder passage). full_content.md sha256 5914ec9b... matches both the file and the response field, and printed title/length match exactly. No local PDF download or local parsing appears anywhere in the transcript or file tree. Docs fetch confirms /v1/extract is GA (only /v1beta/extract is legacy), so no beta/preview marking applies. Minor nit: commands.txt lists the curl script under commands run though the transcript shows only chmod, not execution; the SDK run alone satisfies all criteria.

Attempt record

  1. 0:47Answered from saved key: Parallel API keyno wait
  2. —First success verified

Attempt artifacts saved.