Benchmark report

Task verified

13m 38s elapsed

The saved attempt met the documented success criterion.

Experience score

96/ 100

Agent-Ready
  • AInstall
  • SAuth
  • SExecute
  • SDocs
  • ACurrent
Experience measures friction in the tested task. The verdict says whether it succeeded.
Agent profile for this run: Install A, Auth S, Execute S, Docs S, Current AAInstallSAuthSExecuteSDocsACurrent
Agent profile · this run
Judge's note · what slowed the agent most
Empty response metadata (docs example shows metadata.title) made title extraction brittle, costing two extra live runs before switching to the product's formats.json extraction.

Where the time went

13:39 agent time
  • Starting1:14
  • Research1:17
  • Attempt7:23
  • Verify3:44
13m 38sTotal elapsed
2:14Time to first success
0:00Waited on the owner

Judge reason

Saved code project/scrape_pdf.py POSTs https://arxiv.org/pdf/1706.03762v7 to the documented Context.dev endpoint https://api.context.dev/v1/web/scrape with Bearer auth and sharedParams.parsers.pdf, using only urllib (no local PDF library). Transcript shows a live run returning HTTP 200 (run_stderr.txt 'HTTP 200') and stdout 'Title: Attention Is All You Need / Character length: 39866'. The saved full response evidence/scrape_response.json has markdown.success true, markdown.data of exactly 39,866 chars containing 'Attention Is All You Need' plus real paper body (Scaled Dot-Product, Multi-Head Attention, BLEU, Figure 5 attention visualizations), json.data.title 'Attention Is All You Need' from the product's own extraction, server request_id 07120214-d0f1-4352-942a-267115c7bbb9, cache_metadata miss/age_ms 0, and key_metadata credits_consumed 5 — i.e. a live, uncached, billed server-side parse. parsed.md matches markdown.data byte-for-byte; printed title and length match the response. Docs mark only People Enrich as beta; PDF scrape is GA, so no stop condition applied.

Attempt record

  1. 5:30Answered from saved key: CONTEXT_DEV_API_KEYno wait
  2. —First success verified

Attempt artifacts saved.