Benchmark report
Task verified
9m 16s elapsed
The saved attempt met the documented success criterion.
100/ 100
Agent-Ready- SInstall
- SAuth
- SExecute
- SDocs
- SCurrent
Essentially nothing: docs read, key request, and the live 200 extraction completed in ~50s total with no retries or errors.
Where the time went
9:16 agent time- Starting0:41
- Research0:56
- Attempt5:06
- Verify2:34
Judge reason
Saved code /workspace/home/context-extract/extract.py POSTs https://api.context.dev/v1/web/scrape with Authorization: Bearer <key>, url=https://www.postgresql.org/docs/17/transaction-iso.html, formats.json=true and jsonParams.schema (levels array of objects with name/dirtyRead/nonrepeatableRead/phantomRead/serializationAnomaly) — the product's documented structured-extraction operation, with no local HTML/markdown parsing and no external LLM call. response.json records http_status 200 with a genuine product payload (json.requested true, json.success true, request_id eb571b6b-bc98-4ddf-ba86-d039237a1972, key_metadata.credits_consumed 5 / credits_remaining 4713414, page metadata and headings), and the transcript shows the script actually executed (exit=0, 'HTTP status: 200' in stderr.txt, object echoed to stdout.txt). json.data has exactly four levels matching Table 13.1: Read uncommitted = Allowed, but not in PG/Possible/Possible/Possible; Read committed = Not possible/Possible/Possible/Possible; Repeatable read = Not possible/Not possible/Allowed, but not in PG/Possible; Serializable = Not possible x4. Agent also checked docs.context.dev/scrape/json for beta/preview markings and found none. No fabrication or mocking in evidence.
Attempt record
- 4:07Answered from saved key: CONTEXT_DEV_API_KEYno wait
- —First success verified
Attempt artifacts saved.