Test-Time Compute Scaling with Process Reward Models: Mechanisms, Strategies, and Trade-offs

Understanding Process Reward Models and Step-Level Feedback

Test-Time Compute Scaling (also known as Inference-Time Scaling) allocates additional computational resources, measured in FLOPs, during the generation process rather than relying solely on pre-training scale to improve response accuracy. Evaluating reasoning trajectories during inference depends heavily on the feedback granularity of the verifier model.

Outcome Reward Models (ORMs) evaluate only the final output of a completed response. Because they score solely the terminal state, ORMs suffer from credit assignment problems—frequently rewarding a generation trajectory that happened to arrive at the correct final answer despite relying on flawed intermediate logic.

Process Reward Models (PRMs) address this credit assignment challenge by introducing step-by-step process supervision. Instead of assessing only the end answer, PRMs evaluate each individual intermediate step in a Chain-of-Thought (CoT) trajectory. This step-level feedback provides dense, fine-grained scoring across the response, catching reasoning errors at their point of origin and preventing early mistakes from compounding as the trajectory progresses.

Landmark research by Lightman et al. (2023) established that training reward models with step-by-step process supervision significantly outperforms outcome-based supervision. This concept underpins modern reasoning paradigms more broadly: models such as OpenAI’s o1 and o3 are widely understood to achieve their gains through reinforcement-learned internal reasoning rather than an external verifier, an approach closely related to the Implicit Test-Time Compute Scaling paradigm described below.

Recent developments highlight two distinct architectural approaches to leveraging step-level feedback and compute during inference:

• External Search with PRMs: Using separate, dedicated PRM verifier models to score intermediate steps and steer explicit search algorithms across trajectories.
• Implicit Test-Time Compute Scaling via Reinforcement Learning (RL): Training policy models via RL to produce longer, self-correcting internal Chain-of-Thought (CoT) tokens directly inside a single inference pass without relying on external verifier loops.

Sources: arxiv.org, jhu.edu, youtube.com, github.io, medium.com

Why Step-Level Generative Verification Improves Reasoning

Building on the External Search paradigm introduced above, step-level generative verification provides a critical advancement over standard PRM verifier designs. Traditional Process Reward Models (PRMs) face structural limitations due to their reliance on scalar value predictions without leveraging the generative capabilities of language models, which restricts their process supervision, generalization capabilities, and ability to leverage Test-Time Compute Scaling. Step-level generative verification overcomes these constraints by requiring the model to perform explicit Chain-of-Thought (CoT) reasoning alongside code verification before rendering a judgment for each individual reasoning step.

In frameworks such as GenPRM, step-level generative verification utilizes Relative Progress Estimation (RPE) and a rationale synthesis framework that integrates code verification to produce high-quality process supervision labels and explicit rationale data. This structured step-level rationale generation allows process verification to unlock Test-Time Compute Scaling. Consequently, a 1.5B GenPRM model leverages Test-Time Compute Scaling to outperform GPT-4o, while a 7B GenPRM model surpasses Qwen2.5-Math-PRM-72B on ProcessBench using only 23K training instances from the MATH dataset. Beyond basic step evaluation, generating explicit reasoning and code verification for each step enables the verifier to serve as a critic model for policy model refinement, bridging the operational gap between PRMs and critic models.

Sources: GenPRM: Scaling Test-Time Compute of Process Reward Models via Generative Reasoning

Execution-Based Reward Models for Code Generation Search

While generative verification strengthens step-level scoring, code generation presents a related but distinct challenge for search-based inference. Rather than scoring intermediate reasoning steps directly like a classic PRM, search and verification mechanisms in this domain frequently generate multiple complete candidate solutions and validate each one using execution results from model-generated unit tests as an outcome-based reward signal. However, because models can produce flawed tests with high confidence, these test-based reward signals are often unreliable, degrading the overall quality of candidate verification.

Scaling the number of unit tests provides a positive correlation with reward signal quality, yielding higher performance benefits on more challenging problems. To balance reward precision with execution efficiency during search, a dynamic scaling mechanism can adaptively adjust the quantity of generated unit tests according to problem difficulty. Utilizing CodeRM-8B—a lightweight unit test generator designed for high-quality scaling—alongside dynamic unit test scaling delivers marked accuracy improvements across multiple benchmarks, including performance gains of 18.43% for Llama3-8B and 3.42% for GPT-4o-mini on HumanEval Plus.

Sources: Dynamic Scaling of Unit Tests for Code Reward Modeling

Latency Overhead and Operational Cost Consequences

Although integrating search and verification mechanisms significantly boosts accuracy, scaling compute during inference imposes severe latency overhead and high operational inference costs during active user interactions. To mitigate these execution-time constraints, offline pre-computation approaches such as Sleep-time Compute can reallocate compute demands by allowing models to “think” offline about context prior to query presentation. By anticipating user queries and pre-computing useful quantities ahead of time, this approach significantly decreases the compute required during the actual test-time phase.

Empirical evaluations on Stateful GSM-Symbolic and Stateful AIME show that Sleep-time Compute reduces the amount of Test-Time Compute Scaling needed to achieve equivalent accuracy by approximately 5x. Furthermore, scaling the amount of Sleep-time Compute allocated offline yields direct accuracy improvements, driving gains of up to 13% on Stateful GSM-Symbolic and 18% on Stateful AIME.

To further lower per-query operational costs, Sleep-time Compute can be amortized across multiple requests sharing the same context. In Multi-Query GSM-Symbolic—an extension incorporating multiple related queries per context—amortizing pre-computed quantities across these queries reduces the average compute cost per query by 2.5x. Analysis indicates that the predictability of incoming user queries correlates directly with the effectiveness of Sleep-time Compute in reducing test-time latency and operational overhead. Case study evidence confirms the applicability of this strategy to complex, realistic workloads, including agentic software engineering (SWE) tasks.

Sources: Sleep-time Compute: Beyond Inference Scaling at Test-time

Closing thoughts

Ultimately, the evidence demonstrates that shifting from coarse outcome evaluations to step-level generative verification allows compact models to achieve superior reasoning by systematically catching intermediate errors at their point of origin. However, because scaling search and verification during live inference introduces severe latency and operational cost bottlenecks, relying solely on real-time verifications remains practically constrained. By pairing dynamic step-level feedback with offline strategies like Sleep-time Compute, systems can pre-calculate context and amortize overhead across predictable queries to drastically reduce test-time costs. In my view, the true promise of test-time scaling lies in this precise synergy—balancing fine-grained process supervision with intelligent offline compute allocation to deliver high-accuracy reasoning efficiently.

Frequently Asked Questions

What is the main difference between Outcome Reward Models (ORMs) and Process Reward Models (PRMs)?

Outcome Reward Models evaluate only the final output of a completed response, which can result in rewarding flawed intermediate logic. In contrast, Process Reward Models evaluate each individual intermediate step in a Chain-of-Thought trajectory to catch reasoning errors at their point of origin.

How does step-level generative verification improve reasoning over traditional PRMs?

Traditional PRMs rely on scalar value predictions without using the generative capabilities of language models. Step-level generative verification overcomes this limitation by requiring explicit Chain-of-Thought reasoning alongside code verification before judging each step.

What are two main architectural approaches to using step-level feedback during inference?

The two main approaches are External Search with PRMs, which uses dedicated verifiers to steer explicit search algorithms, and Implicit Test-Time Compute Scaling via RL, which trains policy models to generate self-correcting internal reasoning tokens in a single pass.

How does Sleep-time Compute help mitigate test-time latency overhead?

Sleep-time Compute reallocates compute demands by allowing models to pre-compute context offline before a user presents a query. This reduces the test-time compute required to achieve equivalent accuracy by approximately 5x and lowers per-query costs when amortized across multiple requests.