This repo is an eval harness for the GDP.pdf benchmark, published by Surge AI.

The full dataset is published on Hugging Face. You can use this harness to run GDP.pdf on your own models, or to reproduce the published leaderboard.

This harness is a uv package, built with the Inspect AI framework.

To run an eval: set your provider API keys, then run an inspect eval command:

# can also set these in `.env`
export ANTHROPIC_API_KEY=...
export OPENAI_API_KEY=...
export GEMINI_API_KEY=... 

# evaluate all 100 held-out tasks, no tools
uv run inspect eval src/gdp_pdf/task.py --model anthropic/claude-opus-4-8

# pick the judge, cap tasks, and raise concurrency
uv run inspect eval src/gdp_pdf/task.py \
  --model anthropic/claude-opus-4-20250514 \
  -T judge_model=google/gemini-3.5-flash \
  -T limit=10 \
  --max-connections 8

The scorer reports two metrics per model: all_pass/mean (the headline leaderboard number — every rubric criterion satisfied) and mean_criteria/mean (average fraction of criteria satisfied), comparable to the public leaderboard.

Run multiple epochs with Inspect's score reducers to compute Pass@K and Pass^K in the same run:

uv run inspect eval src/gdp_pdf/task.py --model ... \
  --epochs 5 --epochs-reducer "mean,pass_at_5,at_least_5"

This reports, per metric key: all_pass/mean (Pass@1), all_pass/pass_at_5 (Pass@5, the unbiased Codex-paper estimator), and all_pass/at_least_5 (Pass^5 — all 5 runs pass).

To smoke-test without keys, run the task against a local pack with Inspect's mock model (no network, no judge calls):

# Inspect chdirs into the task file's directory, so pass task_pack as an
# absolute path ($PWD expands before the CLI runs).
uv run inspect eval src/gdp_pdf/task.py \
  -T task_pack="$PWD/task_packs/sample_pdf" \
  --model mockllm/model -T judge_model=mockllm/model

The committed task_packs/sample_pdf/ is a synthetic placeholder task.

Structural tests (no network/keys) run via:

uv run python tests/test_offline.py