RELEASESkyRL v0.5.0 · 2026

SkyRL v0.5.0 Release

Overview

SkyRL v0.5.0 continues our work enabling large scale RL on diverse models, with the following key updates:

  • Model Support: Support for the following new models:
    • Beam-1.0
    • DeepSeek-V4.1-Flash
    • Qwen3.8-Flash-Next
  • RL stability
    • Score Centering
  • Scalability
    • Optimizer Offload to Disk
    • 1M-token context for GLM-5.3-Flash
  • New native On-Policy Distillation (OPD) trainer
  • Vision Language Models: sequence packing, context parallelism, and images in agentic RL

Score Centering

SkyRL v0.5.0 supports score centering to improve RL stability when the rollout and training engines disagree. On GLM-5.3 with NVFP4 W4A4 rollouts and a BF16 trainer, score centering keeps reward high through a six-hour run; without it, reward collapses by step 20. This integration was contributed by Dominic Yurk from Lila Sciences.

In large scale RL runs, the main speed bottleneck is the rate of decoding on inference engines. An easy way to boost throughput drastically on Blackwell systems is to use 4-bit inference for routed experts, but this creates significant train-inference mismatch (TIM) since the trainer still needs to use higher-precision math when computing gradients and weight updates. RL stabilization methods like router replay and importance sampling are insufficient to deal with this level of TIM, leading to unstable training and reward collapse. The addition of Marek and Ryabinin’s score-centering method is the key piece for stabilizing training under high TIM, allowing for robust training performance with 4-bit inference and 16-bit training.

SkyRL computes the score centering correction over the recorded top-k/top-p sampling support, using the sampler’s per-candidate logprobs. This makes the correction exact over the distribution actually sampled, without a tail approximation. It reuses the packed sample-support transport and scoring path, avoids materializing full-vocabulary logits, and works on Megatron and FSDP.

We evaluated score centering on an antibody design task from Lila Sciences using GLM-5.3 with full bf16 training and W4A4 inference for routed experts. The two runs used the same seed and differed only in score centering. Both retained R3, sampler replay (top-k 128, top-p 0.99), and masked importance sampling. The run without score centering collapsed within 20 steps, with exploding entropy and TIM levels and reward going towards 0. In contrast, the run with score centering showed consistent hill climbing for 80 steps with stable entropy and no increase in TIM.

Score centering is automatically enabled when sample-support replay and a compatible sampler-anchored loss are configured. To enable it explicitly with the loss used in this experiment:

trainer.algorithm.policy_loss_type=rollout_is
trainer.algorithm.enable_sample_support_replay=true
trainer.algorithm.enable_score_centering=true

PRs: #2484 (score centering over sampling support), #2608 (automatic enablement).

Score centering on GLM-5.3. NVFP4 W4A4 rollouts and a BF16 trainer, antibody design, seed 42. The run without score centering quickly collapses as TIM blows up, while the score centering run keeps stable TIM as reward climbs. The SC-off run was auto-stopped when the reward collapsed towards 0.

Optimizer Offload to Disk

For training large MoE models like reflection/Beam-1.0-501B-A23B, the optimizer state itself is quite large ~18 Bytes per param (12 B for the fp32 master weights and Adam moments, 4 B for the fp32 gradient copy held by Megatron, and 2 B for the offloaded bf16 weights), i.e. ~9 TB in total. It is common to offload the large optimizer state to CPU to conserve GPU memory. However, even CPU RAM might not be enough and one can run into RAM exhaustion. SkyRL now supports offloading optimizer state to disk. Each rank stores its optimizer state in a file on node-local NVMe and streams it through the GPU one chunk at a time. For training Beam on 8 B200 nodes, each streamed optimizer step takes ~25-50 s per rank, while host RAM usage dropped from ~13.5 TB to <= 3.2 TB (<= 0.4 TB per node). For comparison, the optimizer step with CPU offloading takes ~35s for similar model sizes (300B-500B). Offloading to disk thus enables a ~4x reduction and can be as fast as CPU offloading.

PR: #2490

Long Context

Full fine-tuning of GLM-5.3-Flash now can train at up to 1M tokens per sequence, a 64× increase, through work in four main areas:

  1. DeepSeek Sparse Attention
  2. Activation Recompute and Offloading
  3. Context Parallelism
  4. Avoiding Unnecessary Precision Upcasts

DeepSeek Sparse Attention
 We made sure GLM-5.3-Flash’s DSA layers actually run on a sparse attention kernel. GLM’s query/key vector shapes differ slightly from DeepSeek-V3.2’s, so the fused kernels silently declined them and Megatron fell back to dense attention. Padding GLM’s tensors to the expected shapes fixed this (#2388). We use the TileLang backend over cuDNN/FlashMLA because it pads the number of attention heads far less. We also chunk the DSA indexer so its score matrix never has to be fully materialized (#2413), and split its work across tensor-parallel ranks instead of repeating it on each one (#2422).

Activation Recompute and Offloading
 We turned on full activation recompute, which matters especially for GLM-5.3-Flash, since mHC makes the residual stream 4× wider (#2411). We also release the MoE router state after the forward pass instead of holding on to it through backward (#2433). Finally, we offload each layer’s saved input to CPU memory, using pinned memory so the copies go directly over PCIe instead of being staged from pageable memory through a pinned buffer (#2412).

Context Parallelism
 We added head-wise (Ulysses-style) context parallelism for GLM’s KDA layers, so each GPU runs over the full sequence for a subset of heads (#2418). Instead of exchanging the projected tensors with an all-to-all, we all-gather the much smaller hidden state and have each GPU project only its own heads (#2437).

Avoiding Unnecessary Precision Upcasts
 The mHC projection still computes in FP32, but no longer keeps an FP32 copy of its input for backward. It recomputes the cast instead (#2434). For the MoE combine, when the expert-tensor-parallel group has just two ranks, a single bf16 reduction gives exactly the same result as FP32, so we skip the upcast (#2414).

View the full technical blog here: https://blog.skyrl.ai/blog/1m-context-glm-5-3-flash!

New Model Support

Beam

The new SkyRL v0.5.0 release includes support for training reflection/Beam-1.0-501B-A23B, building on top of our Day 0 support effort. LoRA, full fine-tuning and MXFP8 RL are supported out of the box.

For validating model support, we post-trained the Beam model on the DAPO recipe with vLLM for inference and Megatron for training. LoRA, full fine-tuning and MXFP8 RL achieve similar gains in performance in 50 training steps, or 100 optimizer updates (reasoning minimal), with an average accuracy (avg@8) jump from ~61% to ~79% on AIME25 . Average absolute log-probability difference between Megatron and vLLM stays under 0.01 with Rollout Routing Replay (R3) enabled for LoRA and BF16 full fine-tuning, and under 0.03 for MXFP8 RL.

PRs: #2604, #2618, #2619, #2625

Deepseek v4.1 Flash

SkyRL now supports reinforcement learning on DeepSeek-V4.1-Flash (text only) with the Megatron backend and vLLM inference.

  • LoRA RL on the full model: train rank-32 adapters on attention and expert layers, with adapters synced to vLLM after every step. Validated end to end on GSM8K with 8×B200 nodes.
  • Full fine-tuning: train all weights and sync them to vLLM in the release format. Trainer weights are re-quantized to FP8/MXFP4 and sent in FP8, without a BF16 round trip.
  • Released checkpoint, as is: the trainer loads the released FP8/MXFP4 weights directly, and vLLM serves them with LoRA enabled.
  • DeepSeek-V4: hybrid attention in Megatron and in-place weight sync to vLLM.

PRs: #2567, #2565, #2566, #2551, #2610, #2612, #2646, #2721

Qwen3.8-Flash-Next

SkyRL now supports Qwen3.8-Flash-Next training on the Megatron backend, with both LoRA and full finetuning.

  • Faster sparse attention: we worked with the NVIDIA team to integrate the cuDNN QSA kernels. Compared to our FlexAttention baseline, training is 30% faster at 30K tokens and uses much less memory.
  • Parallelism: the model scales with expert parallelism (EP), for example EP=16 on 32GPUs. EP also shards the frozen 51B-parameter n-gram embedding table across ranks. Tensor parallelism (TP) is supported with sequence parallelism.
  • Long context: with context parallelism, SkyRL trains this model on sequences of up to 262K tokens.

AIME-2024 accuracy, LoRA vs full finetuning

PRs:

  • Qwen3.8-Flash-Next support: #2513
  • Context parallelism up to 262K: #2607

OPD Trainer

Building on top of our previous exploration on on-policy distillation (OPD), SkyRL now supports on-policy distillation (OPD) as an official entrypoint and trainer. We implement sampled-token OPD: the teacher model grades each token the student samples, giving the student dense, per-token signals.

  • Fewer steps than RL: on DAPO-Math-17k, a Qwen3-4B-Base student matches its DAPO-trained teacher’s AIME-2024 accuracy in 20 steps, where DAPO took 90. A Qwen3-1.7B-Base student reaches the 1.7B DAPO baseline within 15 steps.
  • Minimal teacher overhead: the teacher scores each prompt group as soon as they are finished. In our original OPD blog post, the teacher replaced the reference model on the training GPUs and scored the full batch only after generation ended. Now, the teacher runs on its own vLLM engines and generation and scoring is overlapped.
  • Combine task rewards: set trainer.algorithm.opd.use_task_reward=true to combine student rollout results with teacher-scored advantages.
  • Teachers – managed or self-served: SkyRL can use an existing vLLM endpoint as the teacher or launch teacher models itself.

AIME-2024 accuracy, OPD vs DAPO. OPD improves much faster.

In subsequent releases, we plan on supporting more diverse OPD metrics (PR #2624, PR #2632), stabilization strategies (PR #2622), and multi-teacher on-policy distillation (MOPD) (PR #2679).

VLM

This release improves vision-language model (VLM) training on the Megatron backend for Qwen3-VL and Qwen3.5, and adds images to agentic RL.

  • Sequence packing: VLM sequences are now packed, with no padding between sequences of different lengths. On Geometry-3K with Qwen3.5-9B, training throughput goes up by 54%. Set trainer.remove_microbatch_padding=true.
  • Context parallelism (CP): one long sequence can now be split across several GPUs. CP2 gives the same logprobs as CP1 and uses less memory per GPU. Set trainer.policy.megatron_config.context_parallel_size=<n> (example script).
  • Images in agentic RL: skycap now captures agent rollouts that include images, so the trainer sees the same image features as vLLM. We add a SWE-bench Multimodal recipe, where each task comes with screenshots of the bug.

SWE-bench Multimodal mean reward

PRs:

Github Release

Read the full SkyRL v0.5.0 release notes and access the latest code on GitHub.

View SkyRL v0.5.0 on GitHub