Benchmark report

Task verified

13m 18s elapsed

The saved attempt met the documented success criterion.

Experience score

93/ 100

Agent-Ready
  • AInstall
  • SAuth
  • SExecute
  • ADocs
  • ACurrent
Experience measures friction in the tested task. The verdict says whether it succeeded.
Agent profile for this run: Install A, Auth S, Execute S, Docs A, Current AAInstallSAuthSExecuteADocsACurrent
Agent profile · this run
Judge's note · what slowed the agent most
Two re-runs: the doc-style example.com webhook URL was rejected with a misleading 400, and the trigger response carries no run id so runs had to be polled.

Where the time went

13:18 agent time
  • Starting0:01
  • Research0:55
  • Attempt8:34
  • Verify3:48
13m 18sTotal elapsed
2:48Time to first success
0:00Waited on the owner

Judge reason

Saved code (/workspace/home/exa-monitor-task/monitor_lifecycle.py) makes real urllib calls to https://api.exa.ai with x-api-key from a private key file. run_output.txt and full_response.json (identical to evidence-exa-monitor-lifecycle.json) show the live lifecycle: POST /monitors -> HTTP 201 id 01m73k3t4c8swjqsa9z77ejf1d; GET /monitors/{id} -> 200 with query containing https://www.postgresql.org/docs/17/release-17.html, includeDomains=[www.postgresql.org], metadata.targetUrl=target URL and trigger {interval,1h} (the spec's documented minimum: SearchMonitorTrigger 'Minimum interval is 1 hour'), all six read-back checks OK; POST /monitors/{id}/trigger -> 200 {triggered:true}; GET /runs -> run 01m73k3x0y2rkwwp922rkwwp92, then GET /runs/{runId} -> 200 status=completed, durationMs=7099 with real crawled PostgreSQL 17.12 release-note content dated 2026-11-08; DELETE -> 200 and confirming GET -> 404 NOT_FOUND. Cleanup is in a finally block and demonstrably ran on the earlier failed/partial runs (monitor 01m73k30xynwwjpzwc46ypvx3j deleted, 404 confirmed; the first run created nothing and reported nothing to clean up). No beta/preview marking: fetched quickstart and exa-spec.yaml show the marking only on highlights.verbosity/dynamic, which were not used. Nothing mocked or hard-coded.

Attempt record

  1. 5:14Answered from saved key: EXA_API_KEYno wait
  2. —First success verified

Attempt artifacts saved.