Benchmark report
Task verified
4m 40s elapsed
The saved attempt met the documented success criterion.
88/ 100
Agent-Ready- SInstall
- AAuth
- SExecute
- ADocs
- ACurrent
Empty response metadata for the PDF forced a detour into formats.parse (400 parseParams) and writing a title-extraction heuristic over the returned markdown.
Where the time went
4:41 agent time- Research0:17
- Attempt2:58
- Verify1:25
Judge reason
Saved code context-pdf-parse/parse_pdf.py POSTs {"url":"https://arxiv.org/pdf/1706.03762v7","formats":{"markdown":true}} to Context.dev's documented endpoint https://api.context.dev/v1/web/scrape with Bearer auth read from a private key file. response.json is a live 200 response (headers file: X-Request-Id b47accee-b101-42d3-b7f5-855e273c21d8, X-Credits-Used 1, Date Mon 16 Nov 2026 14:10:48 GMT, Cf-Ray, X-Scrape-Ocr-Pages 0) whose markdown.data I independently measured at 39866 characters containing 'Attention Is All You Need' and ending in genuine Figure 5 attention-head text; run-output.txt prints TITLE: Attention Is All You Need and LENGTH: 39866, matching the response exactly. Parsing was done server-side by the product (no local PDF download or PDF library in the code; only urllib for the HTTP call). The title is derived by a generic heuristic over the product-returned text (metadata was empty {} for this PDF), not hard-coded, and the string appears verbatim in the response. A formats.parse probe returned HTTP 400 'parseParams is required' and is saved as response-parse.json; it did not affect the markdown path.
Attempt record
- 0:51Answered from saved key: Context.dev API keyno wait
- —First success verified
Attempt artifacts saved.