Benchmark report

Task verified

4m 40s elapsed

The saved attempt met the documented success criterion.

Experience score

88/ 100

Agent-Ready
  • SInstall
  • AAuth
  • SExecute
  • ADocs
  • ACurrent
Experience measures friction in the tested task. The verdict says whether it succeeded.
Agent profile for this run: Install S, Auth A, Execute S, Docs A, Current ASInstallAAuthSExecuteADocsACurrent
Agent profile · this run
Judge's note · what slowed the agent most
Empty response metadata for the PDF forced a detour into formats.parse (400 parseParams) and writing a title-extraction heuristic over the returned markdown.

Where the time went

4:41 agent time
  • Research0:17
  • Attempt2:58
  • Verify1:25
4m 40sTotal elapsed
0:57Time to first success
0:00Waited on the owner

Judge reason

Saved code context-pdf-parse/parse_pdf.py POSTs {"url":"https://arxiv.org/pdf/1706.03762v7","formats":{"markdown":true}} to Context.dev's documented endpoint https://api.context.dev/v1/web/scrape with Bearer auth read from a private key file. response.json is a live 200 response (headers file: X-Request-Id b47accee-b101-42d3-b7f5-855e273c21d8, X-Credits-Used 1, Date Mon 16 Nov 2026 14:10:48 GMT, Cf-Ray, X-Scrape-Ocr-Pages 0) whose markdown.data I independently measured at 39866 characters containing 'Attention Is All You Need' and ending in genuine Figure 5 attention-head text; run-output.txt prints TITLE: Attention Is All You Need and LENGTH: 39866, matching the response exactly. Parsing was done server-side by the product (no local PDF download or PDF library in the code; only urllib for the HTTP call). The title is derived by a generic heuristic over the product-returned text (metadata was empty {} for this PDF), not hard-coded, and the string appears verbatim in the response. A formats.parse probe returned HTTP 400 'parseParams is required' and is saved as response-parse.json; it did not affect the markdown path.

Attempt record

  1. 0:51Answered from saved key: Context.dev API keyno wait
  2. —First success verified

Attempt artifacts saved.