Nanbeige recurrent policy training

See verification results for the tested source, retained evidence and remaining official-weight GPU gates.

Train Nanbeige4.2-3B at checkpoint revision b82e54bd609793562a75cbf9337970a93369eab5. The provider preserves independent attention histories for each loop, the checkpoint’s head_dim, and its loop-final normalization. Shared physical parameters are optimized once and exported under their native names; the PPO value head is kept out of policy publications.

--algorithm	Policy objective	Critic	Defaults
ppo	Clipped ratio against rollout likelihoods	Yes	GAE, one completion/prompt
grpo	Clipped group-relative advantages	No	Four completions/prompt
dppo	Decoupled PPO: proximal/behavior importance correction	Yes	Recompute proximal scores; correction clipped to [0.5, 2]
flow-dppo	Categorical divergence gate with full-vocabulary KL	Yes	KL budget 0.01, two optimizer steps/rollout

DPPO reuses VIME’s three-policy training path described in Batch size-invariance for policy optimization. Flow-DPPO is a categorical extension of the flow-model algorithm; Nanbeige has a token policy, so this computes full-vocabulary categorical KL rather than a Gaussian transition KL.

Install the pinned native engine and use Megatron 1dcf0dafa884ad52ffb243625717a3471643e087 with VIME’s patch in an isolated environment. The Ray recipe needs one training GPU and one dedicated rollout GPU. JSONL rows contain string prompt and label fields, graded by VIME’s existing math reward. Nanbeige defaults to FP32: its checked FP16 replay exceeded the fixed 0.03 selected-logprob error bound. Keep rollout and training precision identical.

python examples/nanbeige/run.py --algorithm grpo \
  --model /models/Nanbeige4.2-3B --data /data/math.jsonl \
  --output /runs/nanbeige-grpo --updates 3 --recompute \
  --train-env-vars '{"PYTORCH_ALLOC_CONF":"max_split_size_mb:128"}' \
  --model-revision b82e54bd609793562a75cbf9337970a93369eab5 \
  --engine-revision d2a358933393f25d74dd2bdd1068a741cd5d9226

The checked 40 GB A100 PPO run limits large allocator-block splitting with train-env-vars; keep this setting when resuming.

Select each algorithm in a separate output directory. For an interrupted run, add --stop-after 2, then start a fresh process with --resume and without that flag. Keep the total update count, data, precision and batch settings identical. Compare tokens, rewards, traces, actor/critic parameters, optimizer, scheduler, and RNG state against uninterrupted execution. GRPO creates only the actor checkpoint; all-equal rewards correctly produce zero group advantages.

Physical weight publications commit before the next rollout and record their version and content digest. See the shared recipe for packing, optimizer, cache sizing and timing conventions.