GCPO: Diagnosing and Constraining Subspace Geometry in Rollout RL for LLMs

GCPO introduces Geometrically Constrained Policy Optimization, a post-training method that keeps policy updates away from principal subspaces of a frozen base model. This repository provides the open-source GCPO implementation on top of verl, together with basis preparation, data preprocessing, and launch scripts for GRPO- and SDPO-style training.

Figure: Six diagnostic panels summarize stepwise update overlap and validation performance for Qwen3-8B on GSM8K and GLM4-9B on MATH500, showing that higher overlap coincides with worse observed validation checkpoints in these runs.

1. Overview

GCPO adds a geometry-aware constraint to rollout RL by projecting updates away from dominant singular directions of the frozen base model. The repository includes:

• basis construction for the protected subspace;
• dataset loading, splitting, and parquet preprocessing utilities;
• launchers for GRPO and SDPO configurations on local GPUs; and
• the GCPO integration inside the verl training stack.

For environment details and dependency guidance, see INSTALL.md.

2. Installation

GCPO targets Linux with NVIDIA GPUs for practical LLM training. Python 3.10-3.13 is supported by the package metadata, and Python 3.12 is the tested default for both installation paths.

For a native Linux x86_64 environment, install the tested public vLLM 0.12 stack and the matching official FlashAttention wheel before overlaying GCPO:

git clone https://github.com/deepnovacore/GCPO.git
cd GCPO

conda create -n gcpo python=3.12 -y
conda activate gcpo
python -m pip install -r requirements-native.txt

FLASH_ATTN_WHEEL=/tmp/flash_attn-2.8.1+cu12torch2.9cxx11abiTRUE-cp312-cp312-linux_x86_64.whl
curl -fL --retry 5 \
  -o "${FLASH_ATTN_WHEEL}" \
  "https://github.com/Dao-AILab/flash-attention/releases/download/v2.8.1/flash_attn-2.8.1%2Bcu12torch2.9cxx11abiTRUE-cp312-cp312-linux_x86_64.whl"
echo "88ea50d97200b1b0b74f100c5525ae8e1827aae5fd41b488f4fa7489725b4e56  ${FLASH_ATTN_WHEEL}" | sha256sum --check
python -m pip install --no-deps "${FLASH_ATTN_WHEEL}"
python -m pip install -e . --no-deps --no-build-isolation
python -m pip check

The Docker path follows the SDPO environment strategy and starts from NVIDIA's prebuilt vLLM image:

docker build -t gcpo:latest .

Do not mix requirements-native.txt with the NGC image's requirements-ngc.txt, or independently replace GPU-stack packages after installation. See INSTALL.md for both verified version matrices, architecture notes, and verification commands.

On a shared machine, point HF_HOME to a writable location before downloading models or preparing data, for example export HF_HOME=/path/to/huggingface-cache.

3. Data Preparation

GCPO expects each task directory to contain train.json and test.json, followed by train.parquet and test.parquet generated by preprocessing.

3.1 Verified MATH500 example

The following sequence was verified locally for math-ai/math500 with a deterministic 90/10 split (450 train, 50 test):

python data/load_dataset.py \
  --dataset_name math-ai/math500 \
  --output_path datasets/math500/math500.json

python data/split_tasks.py \
  --json_path datasets/math500/math500.json \
  --output_dir datasets/math500 \
  --test_ratio 0.1 \
  --seed 42

python data/preprocess.py --data_source datasets/math500

The preprocessing step writes datasets/math500/train.parquet and datasets/math500/test.parquet, which are the files consumed by the training launcher.

3.2 Custom dataset schema

data/preprocess.py consumes the following JSON fields for each example:

• required: prompt, answer, idx, tests, description, kind, dataset, elo;
• optional: system; and
• training-only optional: achievement_prior.

The tests field is still required for non-code tasks and may contain a placeholder such as "-" when no executable test payload is used by the reward style.

4. Run GCPO

4.1 Prepare the principal-subspace basis

The model choice and TARGET_MODULES must match the later training run.

MODEL_PATH=Qwen/Qwen3-8B \
GCPO_TOPK=16 \
TARGET_MODULES=all-linear \
bash experiments/gcpo/prepare_basis.sh

By default, the basis is written to artifacts/gcpo_basis/<model>/k16/basis.pt. Run this step on a GPU allocation; set DEVICE=cpu only for small models.

4.2 Train with GRPO + GCPO

MODEL_PATH=Qwen/Qwen3-8B \
TASK=datasets/math500 \
CONFIG_NAME=baseline_grpo \
N_GPUS_PER_NODE=4 \
bash experiments/gcpo/train_gcpo.sh

4.3 Train with SDPO + GCPO

MODEL_PATH=Qwen/Qwen3-8B \
TASK=datasets/math500 \
CONFIG_NAME=sdpo \
SDPO_ALPHA=0.5 \
N_GPUS_PER_NODE=4 \
bash experiments/gcpo/train_gcpo.sh

4.4 Main overrides

Variable	Default	Meaning
GCPO_TOPK	16	Number of protected singular directions
LORA_RANK	32	Trainable LoRA rank
LORA_ALPHA	16	LoRA scaling parameter
TRAIN_BATCH_SIZE	32	Prompt batch size
ROLLOUT_N	8	Samples generated per prompt
LR	1e-5	Actor learning rate
TOTAL_TRAINING_STEPS	300	Number of policy updates
OUTPUT_DIR	repository checkpoint directory	Checkpoint destination

5. Docker Quick Start

The root Dockerfile follows SDPO's environment strategy and preserves the matched GPU stack from nvcr.io/nvidia/vllm:25.12.post1-py3. Build the image with:

docker build -t gcpo:latest .

Install the NVIDIA Container Toolkit on the host, then start a GPU container with persistent mounts for model caches, datasets, artifacts, and checkpoints:

docker run --rm -it \
  --gpus all \
  --network host \
  --ipc=host \
  -v /path/to/huggingface-cache:/root/.cache/huggingface \
  -v /path/to/datasets:/app/datasets \
  -v /path/to/artifacts:/app/artifacts \
  -v /path/to/checkpoints:/app/checkpoints \
  gcpo:latest

Replace each /path/to/... entry with a writable host path, then follow Sections 3 and 4 inside /app.

6. Citation

If you find GCPO useful in your research, please cite our paper:

@article{yang2026gcpo,
  title={GCPO: Diagnosing and Constraining Subspace Geometry in Rollout RL for LLMs},
  author={Kai Yang and Jingwei Xu and Wanyu Wang and Kai-Yuan Guo and Zhenbo Yu and Yi Wang and Yu Qiao},
  journal={arXiv preprint arXiv:2608.11674},
  year={2026}
}

7. Acknowledgements

This codebase builds on verl and the SDPO implementation. Their original licenses and notices are preserved in the repository.

Jonas Hübotter et al., “Reinforcement Learning via Self-Distillation,” arXiv:2601.20802, 2026.