Tag: Reinforcement Learning

  • Group Relative Policy Optimization for Efficient Reinforcement Learning in Language Models

    Group Relative Policy Optimization for Efficient Reinforcement Learning in Language Models

    TL;DR

    • Group Relative Policy Optimization (GRPO) eliminates the Critic Model from standard PPO architectures, cutting GPU VRAM usage nearly in half during LLM reinforcement learning.
    • Standard GRPO relies on static sampling and fixed rollouts, which can waste computational resources on easy prompts while under-training difficult reasoning tasks.
    • Combining GRPO with dynamic techniques like Prompt-GDRO and Rollout-GDRO directs compute toward hard tasks, significantly improving reasoning accuracy.

    Understanding Group Relative Policy Optimization (GRPO) in LLM Training

    Reinforcement learning (RL) training methods enhance the alignment and reasoning performance of Large Language Models (LLMs), specifically by improving their capacity to understand human intents, follow user instructions, and strengthen inferential processing.

    Across the broader LLM lifecycle, RL strategies are integrated across several phases: pre-training, alignment fine-tuning, and reinforced reasoning. In particular, RL approaches deployed during the reinforced reasoning phase act as a primary driver for advancing model reasoning limits, with significant focus placed on Reinforcement Learning with Verifiable Rewards (RLVR). Fine-tuning and evaluation within these training frameworks draw upon varied data sources and benchmarks, including human-annotated datasets, AI-assisted preference data, and program-verification-style corpora.

    Sources: Reinforcement Learning Meets Large Language Models: A Survey of Advancements and Applications Across the LLM Lifecycle

    Architectural Shift: Eliminating the Critic Model for Memory Efficiency

    To operationalize these reinforced reasoning phases effectively, attention has increasingly turned to optimizing trainer architectures. In standard Proximal Policy Optimization (PPO), reinforcement learning relies on an Actor-Critic architecture. Fine-tuning a Large Language Model (LLM) Actor Model under PPO requires training a separate Critic Model of a similar size. Running this dedicated value network nearly doubles GPU VRAM requirements and compute overhead during post-training.

    Group Relative Policy Optimization (GRPO) executes an architectural shift by completely eliminating the Critic Model. Rather than using a value network to predict output values, GRPO generates a group of outputs {o1, o2, …, oG} for each input prompt q under the current old policy pi_theta_old. Scores for each output are computed using a reward function—such as rule-based checks for answer correctness—and then standardized across the group to determine relative performance. Removing the Critic Model cuts VRAM usage nearly in half during reinforcement learning training. This direct reduction in memory overhead significantly decreases hardware costs and frees up GPU capacity, enabling models to be trained with higher batch sizes.

    Sources: arxiv.org, substack.com, arxiv.org, huggingface.co, huggingface.co

    Impact on Model Performance, Throughput, and Optimization Limitations

    Despite these architectural memory savings, standard Group Relative Policy Optimization (GRPO) suffers from a structural optimization limitation rooted in static uniformity: it relies on uniform prompt sampling and a fixed number of rollouts per prompt. For heterogeneous, heavy-tailed reasoning datasets, this static approach creates inefficiencies by wasting computational resources on already-solved patterns while under-training the long tail of difficult problems.

    To overcome these compute and optimization bottlenecks, Multi-Adversary Group Distributionally Robust Optimization adapts the training distribution dynamically using an Online Difficulty Classifier that partitions prompts into pass@k difficulty groups: Prompt-GDRO employs an EMA-debiased multiplicative-weights bandit sampler to target the intensive difficulty margin and upweight persistently hard prompt groups without introducing frequency bias, while Rollout-GDRO uses a shadow-price controller guided by a variance-proxy analysis to reallocate rollouts across difficulty groups—aiming for a square-root optimal rollout allocation—to maximize gradient variance reduction on hard tasks under a fixed mean budget.

    Because Rollout-GDRO operates under a fixed mean budget, it dynamically reallocates compute resources in a compute-neutral manner rather than increasing total throughput demands. Qualitative evaluations show that this creates an emergent curriculum, shifting optimization resources toward the evolving reasoning frontier as the model learns. When validated on the DAPO 14.1k dataset using Qwen3-Base models across 1.7B, 4B, and 8B parameter scales, these optimization adjustments yield measurable performance improvements over the standard GRPO baseline, with Prompt-GDRO achieving an average relative gain of +10.6% in pass@8 accuracy and Rollout-GDRO achieving an average relative gain of +10.1% in pass@8 accuracy.

    Sources: Group Distributionally Robust Optimization-Driven Reinforcement Learning for LLM Reasoning

    Closing thoughts

    In summary, by eliminating the memory-heavy Critic Model, Group Relative Policy Optimization offers an invaluable structural solution to the severe VRAM bottlenecks inherent in standard PPO frameworks. However, raw memory efficiency alone falls short when static sampling wastes compute on already-solved problems while under-training the long tail of complex tasks.

    In my assessment, the true breakthrough lies in pairing this critic-free architecture with dynamic techniques like Prompt-GDRO and Rollout-GDRO, which establish an emergent curriculum by targeting compute specifically at the model’s evolving reasoning frontier. Ultimately, these findings demonstrate that maximizing reinforcement learning efficacy requires both stripping away architectural redundancy and intelligently shifting optimization resources toward hard, unresolved prompts.

    Frequently Asked Questions

    What is Group Relative Policy Optimization (GRPO)?

    Group Relative Policy Optimization (GRPO) is a reinforcement learning training method for language models that eliminates the separate Critic Model used in standard PPO architectures. Instead of relying on a value network, GRPO generates a group of outputs per prompt and standardizes their reward scores to measure relative performance.

    How does GRPO reduce memory overhead during training?

    Standard PPO requires training a separate Critic Model of similar size to the Actor Model, which nearly doubles GPU VRAM requirements. By completely removing the Critic Model, GRPO cuts VRAM usage nearly in half and allows models to train with higher batch sizes.

    What limitation exists in standard GRPO?

    Standard GRPO relies on uniform prompt sampling and a fixed number of rollouts per prompt. This static approach can waste computational resources on already-solved patterns while under-training difficult, long-tail reasoning problems.

    How do Prompt-GDRO and Rollout-GDRO address the limitations of GRPO?

    Prompt-GDRO upweights persistently hard prompt groups without frequency bias, while Rollout-GDRO dynamically reallocates rollouts across difficulty groups under a fixed mean budget. Together, they shift optimization resources toward the evolving reasoning frontier, improving accuracy over standard GRPO baselines.

  • Test-Time Compute Scaling with Process Reward Models: Mechanisms, Strategies, and Trade-offs

    Test-Time Compute Scaling with Process Reward Models: Mechanisms, Strategies, and Trade-offs

    TL;DR

    • Process Reward Models evaluate each intermediate step in a Chain-of-Thought trajectory, catching reasoning errors at their point of origin to improve accuracy.
    • Step-level generative verification leverages explicit reasoning and code verification, allowing compact models like GenPRM to outperform much larger baselines.
    • Offline compute strategies like Sleep-time Compute pre-calculate context before queries arrive, significantly reducing test-time latency and operational costs.

    Understanding Process Reward Models and Step-Level Feedback

    Test-Time Compute Scaling (also known as Inference-Time Scaling) allocates additional computational resources, measured in FLOPs, during the generation process rather than relying solely on pre-training scale to improve response accuracy. Evaluating reasoning trajectories during inference depends heavily on the feedback granularity of the verifier model.

    Outcome Reward Models (ORMs) evaluate only the final output of a completed response. Because they score solely the terminal state, ORMs suffer from credit assignment problems—frequently rewarding a generation trajectory that happened to arrive at the correct final answer despite relying on flawed intermediate logic.

    Process Reward Models (PRMs) address this credit assignment challenge by introducing step-by-step process supervision. Instead of assessing only the end answer, PRMs evaluate each individual intermediate step in a Chain-of-Thought (CoT) trajectory. This step-level feedback provides dense, fine-grained scoring across the response, catching reasoning errors at their point of origin and preventing early mistakes from compounding as the trajectory progresses.

    Landmark research by Lightman et al. (2023) established that training reward models with step-by-step process supervision significantly outperforms outcome-based supervision. This concept underpins modern reasoning paradigms more broadly: models such as OpenAI’s o1 and o3 are widely understood to achieve their gains through reinforcement-learned internal reasoning rather than an external verifier, an approach closely related to the Implicit Test-Time Compute Scaling paradigm described below.

    Recent developments highlight two distinct architectural approaches to leveraging step-level feedback and compute during inference:

    • External Search with PRMs: Using separate, dedicated PRM verifier models to score intermediate steps and steer explicit search algorithms across trajectories.
    • Implicit Test-Time Compute Scaling via Reinforcement Learning (RL): Training policy models via RL to produce longer, self-correcting internal Chain-of-Thought (CoT) tokens directly inside a single inference pass without relying on external verifier loops.

    Sources: arxiv.org, jhu.edu, youtube.com, github.io, medium.com

    Why Step-Level Generative Verification Improves Reasoning

    Building on the External Search paradigm introduced above, step-level generative verification provides a critical advancement over standard PRM verifier designs. Traditional Process Reward Models (PRMs) face structural limitations due to their reliance on scalar value predictions without leveraging the generative capabilities of language models, which restricts their process supervision, generalization capabilities, and ability to leverage Test-Time Compute Scaling. Step-level generative verification overcomes these constraints by requiring the model to perform explicit Chain-of-Thought (CoT) reasoning alongside code verification before rendering a judgment for each individual reasoning step.

    In frameworks such as GenPRM, step-level generative verification utilizes Relative Progress Estimation (RPE) and a rationale synthesis framework that integrates code verification to produce high-quality process supervision labels and explicit rationale data. This structured step-level rationale generation allows process verification to unlock Test-Time Compute Scaling. Consequently, a 1.5B GenPRM model leverages Test-Time Compute Scaling to outperform GPT-4o, while a 7B GenPRM model surpasses Qwen2.5-Math-PRM-72B on ProcessBench using only 23K training instances from the MATH dataset. Beyond basic step evaluation, generating explicit reasoning and code verification for each step enables the verifier to serve as a critic model for policy model refinement, bridging the operational gap between PRMs and critic models.

    Sources: GenPRM: Scaling Test-Time Compute of Process Reward Models via Generative Reasoning

    Execution-Based Reward Models for Code Generation Search

    While generative verification strengthens step-level scoring, code generation presents a related but distinct challenge for search-based inference. Rather than scoring intermediate reasoning steps directly like a classic PRM, search and verification mechanisms in this domain frequently generate multiple complete candidate solutions and validate each one using execution results from model-generated unit tests as an outcome-based reward signal. However, because models can produce flawed tests with high confidence, these test-based reward signals are often unreliable, degrading the overall quality of candidate verification.

    Scaling the number of unit tests provides a positive correlation with reward signal quality, yielding higher performance benefits on more challenging problems. To balance reward precision with execution efficiency during search, a dynamic scaling mechanism can adaptively adjust the quantity of generated unit tests according to problem difficulty. Utilizing CodeRM-8B—a lightweight unit test generator designed for high-quality scaling—alongside dynamic unit test scaling delivers marked accuracy improvements across multiple benchmarks, including performance gains of 18.43% for Llama3-8B and 3.42% for GPT-4o-mini on HumanEval Plus.

    Sources: Dynamic Scaling of Unit Tests for Code Reward Modeling

    Latency Overhead and Operational Cost Consequences

    Although integrating search and verification mechanisms significantly boosts accuracy, scaling compute during inference imposes severe latency overhead and high operational inference costs during active user interactions. To mitigate these execution-time constraints, offline pre-computation approaches such as Sleep-time Compute can reallocate compute demands by allowing models to “think” offline about context prior to query presentation. By anticipating user queries and pre-computing useful quantities ahead of time, this approach significantly decreases the compute required during the actual test-time phase.

    Empirical evaluations on Stateful GSM-Symbolic and Stateful AIME show that Sleep-time Compute reduces the amount of Test-Time Compute Scaling needed to achieve equivalent accuracy by approximately 5x. Furthermore, scaling the amount of Sleep-time Compute allocated offline yields direct accuracy improvements, driving gains of up to 13% on Stateful GSM-Symbolic and 18% on Stateful AIME.

    To further lower per-query operational costs, Sleep-time Compute can be amortized across multiple requests sharing the same context. In Multi-Query GSM-Symbolic—an extension incorporating multiple related queries per context—amortizing pre-computed quantities across these queries reduces the average compute cost per query by 2.5x. Analysis indicates that the predictability of incoming user queries correlates directly with the effectiveness of Sleep-time Compute in reducing test-time latency and operational overhead. Case study evidence confirms the applicability of this strategy to complex, realistic workloads, including agentic software engineering (SWE) tasks.

    Sources: Sleep-time Compute: Beyond Inference Scaling at Test-time

    Closing thoughts

    Ultimately, the evidence demonstrates that shifting from coarse outcome evaluations to step-level generative verification allows compact models to achieve superior reasoning by systematically catching intermediate errors at their point of origin. However, because scaling search and verification during live inference introduces severe latency and operational cost bottlenecks, relying solely on real-time verifications remains practically constrained. By pairing dynamic step-level feedback with offline strategies like Sleep-time Compute, systems can pre-calculate context and amortize overhead across predictable queries to drastically reduce test-time costs. In my view, the true promise of test-time scaling lies in this precise synergy—balancing fine-grained process supervision with intelligent offline compute allocation to deliver high-accuracy reasoning efficiently.

    Frequently Asked Questions

    What is the main difference between Outcome Reward Models (ORMs) and Process Reward Models (PRMs)?

    Outcome Reward Models evaluate only the final output of a completed response, which can result in rewarding flawed intermediate logic. In contrast, Process Reward Models evaluate each individual intermediate step in a Chain-of-Thought trajectory to catch reasoning errors at their point of origin.

    How does step-level generative verification improve reasoning over traditional PRMs?

    Traditional PRMs rely on scalar value predictions without using the generative capabilities of language models. Step-level generative verification overcomes this limitation by requiring explicit Chain-of-Thought reasoning alongside code verification before judging each step.

    What are two main architectural approaches to using step-level feedback during inference?

    The two main approaches are External Search with PRMs, which uses dedicated verifiers to steer explicit search algorithms, and Implicit Test-Time Compute Scaling via RL, which trains policy models to generate self-correcting internal reasoning tokens in a single pass.

    How does Sleep-time Compute help mitigate test-time latency overhead?

    Sleep-time Compute reallocates compute demands by allowing models to pre-compute context offline before a user presents a query. This reduces the test-time compute required to achieve equivalent accuracy by approximately 5x and lowers per-query costs when amortized across multiple requests.