Benchmark report
Task verified
13m 38s elapsed
The saved attempt met the documented success criterion.
96/ 100
Agent-Ready- AInstall
- SAuth
- SExecute
- SDocs
- ACurrent
Empty response metadata (docs example shows metadata.title) made title extraction brittle, costing two extra live runs before switching to the product's formats.json extraction.
Where the time went
13:39 agent time- Starting1:14
- Research1:17
- Attempt7:23
- Verify3:44
Judge reason
Saved code project/scrape_pdf.py POSTs https://arxiv.org/pdf/1706.03762v7 to the documented Context.dev endpoint https://api.context.dev/v1/web/scrape with Bearer auth and sharedParams.parsers.pdf, using only urllib (no local PDF library). Transcript shows a live run returning HTTP 200 (run_stderr.txt 'HTTP 200') and stdout 'Title: Attention Is All You Need / Character length: 39866'. The saved full response evidence/scrape_response.json has markdown.success true, markdown.data of exactly 39,866 chars containing 'Attention Is All You Need' plus real paper body (Scaled Dot-Product, Multi-Head Attention, BLEU, Figure 5 attention visualizations), json.data.title 'Attention Is All You Need' from the product's own extraction, server request_id 07120214-d0f1-4352-942a-267115c7bbb9, cache_metadata miss/age_ms 0, and key_metadata credits_consumed 5 — i.e. a live, uncached, billed server-side parse. parsed.md matches markdown.data byte-for-byte; printed title and length match the response. Docs mark only People Enrich as beta; PDF scrape is GA, so no stop condition applied.
Attempt record
- 5:30Answered from saved key: CONTEXT_DEV_API_KEYno wait
- —First success verified
Attempt artifacts saved.