Official code release for ReDiPPO: Reference-Guided Value Calibration and Discrepancy-Aware Token Reweighting for Mathematical Reasoning.

ReDiPPO retains PPO's token-level critic while using a reference answer as training-time privileged information. A reference-guided critic supplies the actor baseline, a standard critic evaluates the same response states without the reference, and their token-level discrepancy reweights the reference-guided advantage. The actor remains prompt-only during training and inference.

For every valid response-token position, ReDiPPO computes

discrepancy = abs(reference_value - standard_value)
weight      = clip(1 + normalize(discrepancy), 0.5, 2.0)
advantage   = whiten(weight * reference_guided_advantage)

Both critics are trained against the same verifier return. Only the reference-guided value is used as the PPO actor baseline; the standard value is retained for the discrepancy calculation.

The public configuration names follow the paper:

.
├── assets/                    Paper figure used in this README
├── critic_pretraining/        Standard/reference-guided critic data and training
├── evaluation/                OpenAI-compatible inference and integer evaluation
├── scripts/
│   ├── data/                  Dataset conversion/filtering utilities
│   └── train/                 ReDiPPO and baseline launchers
├── tests/                     Lightweight tests for ReDiPPO weighting
└── third_party/verl/          Vendored VeRL fork with the ReDiPPO implementation

The main implementation is in:

• third_party/verl/verl/trainer/ppo/core_algos.py: discrepancy weights and weighted GAE support.
• third_party/verl/verl/trainer/ppo/ray_trainer.py: dual-critic dataflow, aligned value inference, critic updates, and monitoring.
• third_party/verl/verl/utils/dataset/rl_dataset.py: reference-answer prompt construction.
• third_party/verl/verl/trainer/config/: public ReDiPPO configuration.

Use a fresh Python environment compatible with the CUDA stack on the training machines. The research code uses the vendored VeRL 0.8.0.dev snapshot.

pip install -r requirements.txt
pip install -e third_party/verl

For critic pretraining:

pip install -r requirements-critic.txt
pip install flash-attn --no-build-isolation

VeRL and the rollout backends are sensitive to CUDA, PyTorch, vLLM, and FlashAttention compatibility. Follow the environment notes in the upstream VeRL documentation when adapting the release to a different cluster.

The paper trains on DAPO-17K and an integer-answer subset of DeepMath-103K. Dataset files are not redistributed here; users must follow the licenses of the original datasets.

The VeRL parquet rows must contain:

• a prompt field compatible with the configured VeRL dataset loader;
• a reference answer, addressed by data.reference_answer_key=reward_model.ground_truth;
• the normal VeRL metadata required by the selected reward manager.

The default reference template is exactly the one used in the paper:

The ground truth answer is {answer}.

The release includes two data utilities:

python scripts/data/filter_integer_answers.py --help
python scripts/data/parquet_to_jsonl.py --help

The launchers expect the following layout by default:

${DATA_ROOT}/
├── dapo_test.parquet
├── qwen3-4b-instruct/train_dapo_deepmath.parquet
├── qwen3-4b-thinking/train_dapo_deepmath.parquet
└── olmo3-instruct/train_dapo_deepmath.parquet

Paths can be changed directly in the corresponding launcher.

ReDiPPO requires two value-model checkpoints initialized from the same policy:

1. the standard critic, trained on question + response;
2. the reference-guided critic, trained on question + reference answer + response.

Given inference records containing question, model_output, final_correct, and answer, build the two training files:

python critic_pretraining/scripts/build_standard_critic_data.py \
  --input_file /path/to/inference.jsonl \
  --output_file /path/to/standard_critic.jsonl \
  --model_name_or_path /path/to/policy \
  --filter_balanced \
  --overwrite

python critic_pretraining/scripts/build_reference_critic_data.py \
  --input_file /path/to/inference.jsonl \
  --output_file /path/to/reference_critic.jsonl \
  --model_name_or_path /path/to/policy \
  --filter_balanced \
  --overwrite

Pretrain each critic for two epochs:

MODEL_NAME_OR_PATH=/path/to/policy \
TRAIN_FILE=/path/to/standard_critic.jsonl \
OUTPUT_DIR=/path/to/qwen3-4b-instruct-standard \
NGPUS=8 \
bash critic_pretraining/train.sh

MODEL_NAME_OR_PATH=/path/to/policy \
TRAIN_FILE=/path/to/reference_critic.jsonl \
OUTPUT_DIR=/path/to/qwen3-4b-instruct-reference \
NGPUS=8 \
bash critic_pretraining/train.sh

See critic_pretraining/README.md for the data schema and optional settings.

The release provides matched ReDiPPO, PPO, DAPO, and GSPO launchers for all three paper backbones:

Example:

MODEL_ROOT=/path/to/pretrained_models \
DATA_ROOT=/path/to/data \
OUTPUT_ROOT=/path/to/outputs \
STANDARD_CRITIC_PATH=/path/to/qwen3-4b-instruct-standard \
REFERENCE_CRITIC_PATH=/path/to/qwen3-4b-instruct-reference \
bash scripts/train/qwen3-4b-instruct-redippo.sh redippo qwen3-4b-instruct

The scripts reproduce the paper defaults:

• global batch size: 512;
• responses per prompt: 8;
• actor/critic learning rates: 1e-6 / 5e-6;
• gamma = lambda = 1;
• asymmetric PPO clip range: [0.8, 1.28];
• discrepancy-weight range: [0.5, 2.0];
• token-level discrepancy weights applied before masked whitening.

The checked-in launchers default to 2 nodes × 8 GPUs. Adjust NNODES, NGPUS, micro-batch sizes, token limits, and offloading settings for the available hardware.

evaluation/infer.py is a high-throughput client for an OpenAI-compatible model server. evaluation/evaluate_int.py implements the integer-answer evaluation used for the paper's mathematical benchmarks.

python evaluation/infer.py \
  --input_file /path/to/benchmark.jsonl \
  --output_file /path/to/predictions.jsonl \
  --host 127.0.0.1 \
  --port 8000 \
  --temperature 0.6 \
  --num_repeats 32 \
  --max_new_tokens 8192

python evaluation/evaluate_int.py \
  --pred_file /path/to/predictions.jsonl \
  --repeat_k 32

See evaluation/README.md for the expected fields.

Released under the Apache License 2.0. See LICENSE and NOTICE. The vendored VeRL notice is preserved under third_party/verl/.