Fixed-depth recurrent policy training

The shared launcher runs VIME’s standard Ray entry with native vLLM-RLT rollout. Nanbeige documents its model-specific command and all four algorithms. Ouro also uses this entry with --model /models/Ouro-1.4B and checkpoint revision 574fa66cb8bf5abdc979642d01cf2b79b16bfab1.

Use --algorithm ppo, grpo, dppo, or flow-dppo. PPO and GRPO reuse the existing clipped policy loss. DPPO keeps rollout behavior scores separate from proximal scores recomputed by the training provider, and applies VIME’s existing bounded importance correction. Flow-DPPO freezes full-vocabulary old-policy scores and applies the categorical divergence gate. Actor/critic roles use GAE for PPO, DPPO and Flow-DPPO; GRPO uses an actor and prompt-group reward normalization.

Install the pinned RLT engine and Megatron 1dcf0dafa884ad52ffb243625717a3471643e087 with VIME’s patch in an isolated environment. Use an existing two-GPU Ray cluster; --ray-address host:port selects its address. The launcher propagates its Python executable and source path to workers. Math JSONL rows need string prompt and label fields, for example:

{"prompt":"Calculate 7-4. Reply with only \\boxed{answer}.\n###Response\n","label":"3"}

Defaults use SGD without momentum, FP32 gradient accumulation, four prompts, 128 prompt tokens, 48 response tokens and full-vocabulary sampling. Nanbeige and Ouro use FP32. --precision fp16 selects fixed loss scale 128. Local MCore checkpointing restores optimizer parameter groups, and FP16 master parameters and loss scale. --recompute checkpoints each physical block. --rlt-cuda-graphs selects the engine’s graph path; eager execution is the default. Cache capacity accounts for all recurrence planes of two active sequences.

--rollout-batch-size counts prompts and --n-samples-per-prompt counts completions. GRPO defaults to four completions and requires at least two. Its reward normalizer runs before samples are packed; a second batch-wide advantage whitening is omitted. PPO/DPPO use one optimizer step per rollout. Flow-DPPO uses two steps, each with half the completion batch. Extra arguments pass through to VIME.

Use a separate output directory with --stop-after 2, then a fresh process with --resume for interrupted/resumed comparisons against three uninterrupted updates. Actor and critic checkpoints are separate; GRPO saves only the actor. The dataset cursor and policy version continue, and restored weights publish before generation. Compare full model/optimizer/scheduler/RNG state, tokens, rewards and traces. The launcher enables deterministic training and passes the cuBLAS workspace and NCCL algorithm settings to Ray workers for this comparison. Report end-to-end timing including rollout, rewards, optimization, full publications and checkpoints, with source pins, dtype and resource contention.