Category: AI/ML Concepts

  • Byte Latent Transformer: Replacing Tokenizers with Dynamic Entropy-Based Patching

    Byte Latent Transformer: Replacing Tokenizers with Dynamic Entropy-Based Patching

    TL;DR

    • The Byte Latent Transformer introduces a tokenizer-free architecture that overcomes the noise brittleness and vocabulary biases of traditional subword tokenization.
    • It processes raw byte streams efficiently by dynamically segmenting bytes into patches based on next-byte entropy.
    • BLT matches or exceeds the performance of traditional tokenizer-based models like LLaMA at scales up to 8 billion parameters.

    Introduction: The Shift from Fixed Tokenizers to Byte-Level Modeling

    Traditional large language models depend heavily on heuristic Subword Tokenization algorithms, such as Byte-Pair Encoding (BPE), to segment text into static subword vocabularies. While widely adopted, fixed Subword Tokenization introduces fundamental structural vulnerabilities:

    • Brittleness to Noise and Manipulation: Models exhibit high sensitivity to typos, spelling errors, and character-level perturbations, while struggling on character-level manipulation and arithmetic tasks.
    • Vocabulary and Representation Issues: Fixed vocabularies lead to out-of-vocabulary fragmentation and create structural biases favoring English and high-resource scripts over low-resource languages.

    Operating directly on raw byte streams eliminates the need for tokenizers, but naive Byte-Level Language Model design introduces severe computational bottlenecks. Expanding text directly into raw byte sequences dramatically inflates sequence lengths, imposing extreme quadratic computational overhead on standard Transformer self-attention mechanisms.

    To overcome both the rigidity of subword tokenizers and the computational expense of naive byte sequences, researchers from Meta FAIR, the University of Washington, and the University of Chicago (Artidoro Pagnoni, Ram Pasunuru, Pedro Rodriguez, John Nguyen, Benjamin Muller, Margaret Li, Chunting Zhou, Lili Yu, Jason Weston, Luke Zettlemoyer, Gargi Ghosh, Mike Lewis, Ari Holtzman, and Srinivasan Iyer) introduced the Byte Latent Transformer (BLT). BLT represents the first compute- and FLOP-controlled scaling study demonstrating that a tokenizer-free Byte-Level Language Model can match or exceed the performance of standard tokenizer-based models, such as LLaMA baselines, across scales up to 8 billion parameters and trillions of training bytes.

    Sources: youtube.com, arxiv.org, arxiv.org, openreview.net, aclanthology.org

    Mechanism: Dynamically Segmenting Patches via Next-Byte Entropy

    At the core of this breakthrough is the Byte Latent Transformer (BLT) architecture’s approach to sequence structure: raw bytes are segmented into dynamically sized patches through Dynamic Entropy-Based Patching, serving as the primary units of computation and bypassing the need for Subword Tokenization. Patch segmentation is driven by Next-Byte Entropy predicted by an Entropy Model, which enables the model to dynamically allocate more compute and model capacity where increased data complexity demands it.

    When the underlying data is predictable, Dynamic Entropy-Based Patching constructs longer patches. This entropy-based segmentation enhances both training and inference efficiency, allowing BLT to simultaneously scale model size and patch length under fixed inference costs.

    Sources: Byte Latent Transformer: Patches Scale Better Than Tokens

    Closing thoughts

    Ultimately, by tying sequence segmentation directly to informational entropy rather than static vocabulary heuristics, the Byte Latent Transformer successfully resolves the long-standing computational bottleneck of naive byte-level processing. In my view, the fact that BLT matches or exceeds standard baselines like LLaMA at scales up to 8 billion parameters demonstrates that the structural brittleness and vocabulary biases of BPE are no longer necessary compromises for scalable architectures. Dynamically scaling patch lengths according to data predictability proves to be a well-reasoned solution, allowing byte-level modeling to achieve character-level robustness while strictly maintaining compute and inference efficiency.

    Frequently Asked Questions

    What are the main drawbacks of traditional subword tokenization methods like BPE?

    Fixed subword tokenization introduces brittleness to noise, typos, and character-level perturbations, while causing out-of-vocabulary fragmentation. It also struggles with character manipulation tasks and creates structural biases favoring English and high-resource scripts over low-resource languages.

    Why is naive byte-level modeling computationally inefficient?

    Directly expanding text into raw byte sequences dramatically inflates sequence lengths. This imposes extreme quadratic computational overhead on standard Transformer self-attention mechanisms.

    How does the Byte Latent Transformer (BLT) process byte sequences efficiently?

    BLT uses Dynamic Entropy-Based Patching driven by next-byte entropy predicted by an Entropy Model. It constructs longer patches when data is predictable, dynamically allocating more compute and capacity where data complexity demands it.

    How does the performance of BLT compare to standard tokenizer-based models?

    BLT matches or exceeds the performance of standard tokenizer-based models, such as LLaMA baselines, across scales up to 8 billion parameters and trillions of training bytes.

  • Titans: Learning to Memorize at Test Time with Neural Long-Term Memory

    Titans: Learning to Memorize at Test Time with Neural Long-Term Memory

    TL;DR

    • Titans introduces a dual-memory architecture combining attention for short-term dependencies with a Neural Long-Term Memory module for persistent historical context.
    • The framework resolves trade-offs between quadratic attention and lossy recurrent models through three variants: Memory as Context, Memory as Gate, and Memory as Layer.
    • Titans scales to context windows exceeding 2 million tokens and achieves superior performance across language, genomics, time series, and Needle-in-a-Haystack benchmarks.

    Introduction to Titans and Test-Time Neural Memory

    Titans introduces a dual-memory architectural paradigm that combines attention with a dedicated Neural Long-Term Memory module. In this framework, attention acts as short-term memory, leveraging its accurate token dependency modeling over a limited context window, while the Neural Long-Term Memory module acts as persistent memory by learning to memorize historical context. This Neural Long-Term Memory allows the system to retain and utilize long-past information to assist attention on the current context, while maintaining the advantages of fast parallelizable training and fast inference.

    Built upon the integration of these two complementary modules, the Titans family features three architectural variants designed to incorporate memory effectively: Memory as Context (MAC), Memory as Gate (MAG), and Memory as Layer (MAL). Across evaluations spanning language modeling, common-sense reasoning, genomics, and time series benchmarks, Titans demonstrates greater effectiveness than Transformers and modern linear recurrent models. Furthermore, the architecture scales to context window sizes exceeding 2M tokens while achieving higher accuracy on Needle-in-a-Haystack tasks relative to baseline models.

    Sources: Titans: Learning to Memorize at Test Time

    Architectural Paradigms: Comparing Attention and Recurrent Memory Trade-Offs

    To understand the motivation behind this architecture, it is helpful to examine modern sequence modeling, which has long been characterized by a fundamental tension between attention-based architectures and recurrent models. On one hand, Transformers excel at capturing precise, fine-grained, and direct dependencies between tokens across a context window. However, self-attention incurs a quadratic computational complexity of O(N^2) with respect to sequence length, resulting in extreme memory footprints and computational bottlenecks that severely restrict the size of usable context windows.

    In contrast, Recurrent Neural Networks (RNNs) and linear recurrent models offer linear computational complexity of O(N) and fast inference by compressing historical context into a fixed-size hidden state. However, this fixed-capacity compression introduces significant information loss and degrades associative recall over extensive contexts.

    Titans reconciles these conflicting paradigms by structuring memory hierarchically. Rather than relying entirely on quadratic self-attention or static hidden-vector compression, the architecture assigns short-term, high-resolution local dependencies to an attention mechanism while offloading past historical context to a dynamic Neural Long-Term Memory module. This design retains the fast inference and linear compute scaling of recurrent models without sacrificing the associative fidelity required for long-context sequence modeling.

    Sources: arxiv.org, youtube.com, medium.com, openreview.net, huggingface.co

    Closing thoughts

    Ultimately, by reframing sequence modeling as a hierarchical dual-memory system rather than a single compromised mechanism, Titans convincingly resolves the structural tension between quadratic attention bottlenecks and lossy recurrent compression. In our view, assigning persistent historical context to an active neural long-term memory while reserving standard attention for high-resolution local dependencies is the key architectural insight that allows the model to scale past two million tokens without degrading retrieval fidelity.

    Furthermore, the demonstrated gains across domains as varied as genomics, time series, and language benchmarks suggest that modular memory integration—whether via context, gating, or dedicated layers—is fundamentally more effective than forcing a uniform mechanism to handle both immediate and distant dependencies. In doing so, Titans proves that sequence models can achieve fast inference and linear scaling without sacrificing the associative accuracy essential for vast context windows.

    Frequently Asked Questions

    What is the dual-memory architecture used in Titans?

    Titans combines an attention mechanism that serves as short-term memory with a dedicated Neural Long-Term Memory module for persistent memory. This design allows the model to retain past information to support attention on the current context while maintaining fast parallelizable training and fast inference.

    What are the three architectural variants of Titans?

    The Titans family includes three architectural variants: Memory as Context (MAC), Memory as Gate (MAG), and Memory as Layer (MAL).

    How does Titans resolve the trade-offs between Transformers and recurrent models?

    Titans organizes memory hierarchically by assigning short-term, high-resolution local dependencies to attention while offloading past historical context to a Neural Long-Term Memory module. This enables linear compute scaling and fast inference without the information loss typical of static recurrent compression.

    What context window size can Titans achieve?

    Titans scales to context window sizes exceeding 2 million tokens while delivering higher accuracy on Needle-in-a-Haystack tasks compared to baseline models.

  • Group Relative Policy Optimization for Efficient Reinforcement Learning in Language Models

    Group Relative Policy Optimization for Efficient Reinforcement Learning in Language Models

    TL;DR

    • Group Relative Policy Optimization (GRPO) eliminates the Critic Model from standard PPO architectures, cutting GPU VRAM usage nearly in half during LLM reinforcement learning.
    • Standard GRPO relies on static sampling and fixed rollouts, which can waste computational resources on easy prompts while under-training difficult reasoning tasks.
    • Combining GRPO with dynamic techniques like Prompt-GDRO and Rollout-GDRO directs compute toward hard tasks, significantly improving reasoning accuracy.

    Understanding Group Relative Policy Optimization (GRPO) in LLM Training

    Reinforcement learning (RL) training methods enhance the alignment and reasoning performance of Large Language Models (LLMs), specifically by improving their capacity to understand human intents, follow user instructions, and strengthen inferential processing.

    Across the broader LLM lifecycle, RL strategies are integrated across several phases: pre-training, alignment fine-tuning, and reinforced reasoning. In particular, RL approaches deployed during the reinforced reasoning phase act as a primary driver for advancing model reasoning limits, with significant focus placed on Reinforcement Learning with Verifiable Rewards (RLVR). Fine-tuning and evaluation within these training frameworks draw upon varied data sources and benchmarks, including human-annotated datasets, AI-assisted preference data, and program-verification-style corpora.

    Sources: Reinforcement Learning Meets Large Language Models: A Survey of Advancements and Applications Across the LLM Lifecycle

    Architectural Shift: Eliminating the Critic Model for Memory Efficiency

    To operationalize these reinforced reasoning phases effectively, attention has increasingly turned to optimizing trainer architectures. In standard Proximal Policy Optimization (PPO), reinforcement learning relies on an Actor-Critic architecture. Fine-tuning a Large Language Model (LLM) Actor Model under PPO requires training a separate Critic Model of a similar size. Running this dedicated value network nearly doubles GPU VRAM requirements and compute overhead during post-training.

    Group Relative Policy Optimization (GRPO) executes an architectural shift by completely eliminating the Critic Model. Rather than using a value network to predict output values, GRPO generates a group of outputs {o1, o2, …, oG} for each input prompt q under the current old policy pi_theta_old. Scores for each output are computed using a reward function—such as rule-based checks for answer correctness—and then standardized across the group to determine relative performance. Removing the Critic Model cuts VRAM usage nearly in half during reinforcement learning training. This direct reduction in memory overhead significantly decreases hardware costs and frees up GPU capacity, enabling models to be trained with higher batch sizes.

    Sources: arxiv.org, substack.com, arxiv.org, huggingface.co, huggingface.co

    Impact on Model Performance, Throughput, and Optimization Limitations

    Despite these architectural memory savings, standard Group Relative Policy Optimization (GRPO) suffers from a structural optimization limitation rooted in static uniformity: it relies on uniform prompt sampling and a fixed number of rollouts per prompt. For heterogeneous, heavy-tailed reasoning datasets, this static approach creates inefficiencies by wasting computational resources on already-solved patterns while under-training the long tail of difficult problems.

    To overcome these compute and optimization bottlenecks, Multi-Adversary Group Distributionally Robust Optimization adapts the training distribution dynamically using an Online Difficulty Classifier that partitions prompts into pass@k difficulty groups: Prompt-GDRO employs an EMA-debiased multiplicative-weights bandit sampler to target the intensive difficulty margin and upweight persistently hard prompt groups without introducing frequency bias, while Rollout-GDRO uses a shadow-price controller guided by a variance-proxy analysis to reallocate rollouts across difficulty groups—aiming for a square-root optimal rollout allocation—to maximize gradient variance reduction on hard tasks under a fixed mean budget.

    Because Rollout-GDRO operates under a fixed mean budget, it dynamically reallocates compute resources in a compute-neutral manner rather than increasing total throughput demands. Qualitative evaluations show that this creates an emergent curriculum, shifting optimization resources toward the evolving reasoning frontier as the model learns. When validated on the DAPO 14.1k dataset using Qwen3-Base models across 1.7B, 4B, and 8B parameter scales, these optimization adjustments yield measurable performance improvements over the standard GRPO baseline, with Prompt-GDRO achieving an average relative gain of +10.6% in pass@8 accuracy and Rollout-GDRO achieving an average relative gain of +10.1% in pass@8 accuracy.

    Sources: Group Distributionally Robust Optimization-Driven Reinforcement Learning for LLM Reasoning

    Closing thoughts

    In summary, by eliminating the memory-heavy Critic Model, Group Relative Policy Optimization offers an invaluable structural solution to the severe VRAM bottlenecks inherent in standard PPO frameworks. However, raw memory efficiency alone falls short when static sampling wastes compute on already-solved problems while under-training the long tail of complex tasks.

    In my assessment, the true breakthrough lies in pairing this critic-free architecture with dynamic techniques like Prompt-GDRO and Rollout-GDRO, which establish an emergent curriculum by targeting compute specifically at the model’s evolving reasoning frontier. Ultimately, these findings demonstrate that maximizing reinforcement learning efficacy requires both stripping away architectural redundancy and intelligently shifting optimization resources toward hard, unresolved prompts.

    Frequently Asked Questions

    What is Group Relative Policy Optimization (GRPO)?

    Group Relative Policy Optimization (GRPO) is a reinforcement learning training method for language models that eliminates the separate Critic Model used in standard PPO architectures. Instead of relying on a value network, GRPO generates a group of outputs per prompt and standardizes their reward scores to measure relative performance.

    How does GRPO reduce memory overhead during training?

    Standard PPO requires training a separate Critic Model of similar size to the Actor Model, which nearly doubles GPU VRAM requirements. By completely removing the Critic Model, GRPO cuts VRAM usage nearly in half and allows models to train with higher batch sizes.

    What limitation exists in standard GRPO?

    Standard GRPO relies on uniform prompt sampling and a fixed number of rollouts per prompt. This static approach can waste computational resources on already-solved patterns while under-training difficult, long-tail reasoning problems.

    How do Prompt-GDRO and Rollout-GDRO address the limitations of GRPO?

    Prompt-GDRO upweights persistently hard prompt groups without frequency bias, while Rollout-GDRO dynamically reallocates rollouts across difficulty groups under a fixed mean budget. Together, they shift optimization resources toward the evolving reasoning frontier, improving accuracy over standard GRPO baselines.

  • SimPO: Reference-Free Preference Optimization for Large Language Models

    SimPO: Reference-Free Preference Optimization for Large Language Models

    TL;DR

    • Developed by researchers at Princeton and UVA, SimPO is a reference-free preference optimization method that eliminates the need for a frozen reference model in LLM alignment.
    • By utilizing sequence-average log probabilities and a target reward margin, SimPO aligns training objectives with inference while reducing GPU memory usage and training overhead.
    • SimPO consistently outperforms DPO across standard benchmarks like AlpacaEval 2 and Arena-Hard while mitigating token verbosity and length bias.

    Introduction to SimPO: A Simple Reference-Free Approach

    SimPO (Simple Preference Optimization) is a reference-free preference optimization method designed to align large language models (LLMs) directly with human preferences. Introduced by researchers Yu Meng, Mengzhou Xia, and Danqi Chen from Princeton University and the University of Virginia, the approach was published in the paper SimPO: Simple Preference Optimization with a Reference-Free Reward (arXiv:2405.14734) and accepted as a NeurIPS 2024 paper. Official code implementations and open-source model checkpoints are hosted under the Princeton NLP organization.

    SimPO was motivated by the operational and theoretical limitations of Direct Preference Optimization (DPO). Although DPO simplified Reinforcement Learning from Human Feedback (RLHF) by training policy models directly on human preference pairs without a separate reward model, DPO still relies on a Reference Model—a frozen copy of the base supervised fine-tuned (SFT) model—to constrain policy drift using Kullback-Leibler (KL) divergence. This design introduces two key challenges:

    • Training-Inference Discrepancy: DPO calculates Implicit Reward based on a log-likelihood ratio relative to the frozen Reference Model. Because LLMs generate text during inference based directly on sequence log-probabilities rather than relative likelihood ratios, this ratio-based Implicit Reward creates a misalignment between the training objective and actual generation behavior.
    • Compute and Memory Inefficiency: Keeping a second frozen Reference Model in GPU memory throughout training requires extra forward passes, increasing VRAM usage and compute overhead.

    SimPO resolves these issues by eliminating the Reference Model entirely from the alignment objective. Removing the Reference Model reduces memory consumption and speeds up training, enabling researchers and practitioners to fine-tune larger models on smaller GPU clusters. Furthermore, SimPO aligns the training objective with inference generation while avoiding the token verbosity and length bias frequently observed in traditional RLHF and DPO implementations. Across standard alignment evaluations—such as AlpacaEval 2, Arena-Hard, and MT-Bench—SimPO consistently outperforms DPO and its variants across multiple model families, including Llama 3, Gemma 2, and Mistral.

    Sources: arxiv.org, github.io, openreview.net, arxiv.org, huggingface.co

    Algorithmic Design: Sequence-Average Probabilities and Target Margins

    To address these limitations, the core algorithmic design of SimPO relies on using the Average Log Probability of a sequence as its Implicit Reward formulation. By utilizing sequence Average Log Probability, the Implicit Reward directly aligns with the language model’s generation process. Crucially, this reward formulation eliminates the requirement for a Reference Model during training, making the optimization process both compute and memory efficient.

    Additionally, SimPO modifies the traditional Bradley-Terry objective by incorporating a target reward margin. This target margin explicitly enforces a larger margin between the Implicit Rewards of the winning and dispreferred losing responses, driving a distinct separation between candidate outputs to further enhance the algorithm’s performance.

    Sources: SimPO: Simple Preference Optimization with a Reference-Free Reward

    Closing thoughts

    Building on these algorithmic refinements, SimPO demonstrates that the architectural reliance on a frozen reference model was an unnecessary hurdle in preference optimization. By aligning the implicit reward directly with sequence-average log probabilities and enforcing a target margin, the algorithm logically resolves the mismatch between how models are trained and how they actually generate text. In my view, its primary strength lies in achieving this conceptual elegance alongside immediate practical gains—slashing GPU memory requirements while simultaneously curbing the length bias that plagues traditional RLHF. As evidenced by its superior performance across benchmarks like AlpacaEval 2 and Arena-Hard, SimPO proves that streamlining alignment mechanics yields a significantly more resource-efficient workflow without compromising on output quality.

    Frequently Asked Questions

    What is SimPO and who developed it?

    SimPO (Simple Preference Optimization) is a reference-free preference optimization method designed to align large language models directly with human preferences. It was introduced by researchers Yu Meng, Mengzhou Xia, and Danqi Chen from Princeton University and the University of Virginia.

    How does SimPO improve upon Direct Preference Optimization (DPO)?

    Unlike DPO, SimPO completely eliminates the need for a frozen reference model, which slashes GPU memory usage and speeds up training. It also resolves the discrepancy between training objectives and inference generation while curbing length bias.

    What are the core algorithmic components of SimPO?

    SimPO uses the sequence-average log probability of an output as its implicit reward formulation. It also modifies the Bradley-Terry objective by adding a target reward margin to enforce a clear separation between preferred and dispreferred responses.

    How does SimPO perform on standard language model benchmarks?

    SimPO consistently outperforms DPO and its variants across standard alignment evaluations, including AlpacaEval 2, Arena-Hard, and MT-Bench. These gains are demonstrated across multiple model families, such as Llama 3, Gemma 2, and Mistral.

  • Test-Time Compute Scaling with Process Reward Models: Mechanisms, Strategies, and Trade-offs

    Test-Time Compute Scaling with Process Reward Models: Mechanisms, Strategies, and Trade-offs

    TL;DR

    • Process Reward Models evaluate each intermediate step in a Chain-of-Thought trajectory, catching reasoning errors at their point of origin to improve accuracy.
    • Step-level generative verification leverages explicit reasoning and code verification, allowing compact models like GenPRM to outperform much larger baselines.
    • Offline compute strategies like Sleep-time Compute pre-calculate context before queries arrive, significantly reducing test-time latency and operational costs.

    Understanding Process Reward Models and Step-Level Feedback

    Test-Time Compute Scaling (also known as Inference-Time Scaling) allocates additional computational resources, measured in FLOPs, during the generation process rather than relying solely on pre-training scale to improve response accuracy. Evaluating reasoning trajectories during inference depends heavily on the feedback granularity of the verifier model.

    Outcome Reward Models (ORMs) evaluate only the final output of a completed response. Because they score solely the terminal state, ORMs suffer from credit assignment problems—frequently rewarding a generation trajectory that happened to arrive at the correct final answer despite relying on flawed intermediate logic.

    Process Reward Models (PRMs) address this credit assignment challenge by introducing step-by-step process supervision. Instead of assessing only the end answer, PRMs evaluate each individual intermediate step in a Chain-of-Thought (CoT) trajectory. This step-level feedback provides dense, fine-grained scoring across the response, catching reasoning errors at their point of origin and preventing early mistakes from compounding as the trajectory progresses.

    Landmark research by Lightman et al. (2023) established that training reward models with step-by-step process supervision significantly outperforms outcome-based supervision. This concept underpins modern reasoning paradigms more broadly: models such as OpenAI’s o1 and o3 are widely understood to achieve their gains through reinforcement-learned internal reasoning rather than an external verifier, an approach closely related to the Implicit Test-Time Compute Scaling paradigm described below.

    Recent developments highlight two distinct architectural approaches to leveraging step-level feedback and compute during inference:

    • External Search with PRMs: Using separate, dedicated PRM verifier models to score intermediate steps and steer explicit search algorithms across trajectories.
    • Implicit Test-Time Compute Scaling via Reinforcement Learning (RL): Training policy models via RL to produce longer, self-correcting internal Chain-of-Thought (CoT) tokens directly inside a single inference pass without relying on external verifier loops.

    Sources: arxiv.org, jhu.edu, youtube.com, github.io, medium.com

    Why Step-Level Generative Verification Improves Reasoning

    Building on the External Search paradigm introduced above, step-level generative verification provides a critical advancement over standard PRM verifier designs. Traditional Process Reward Models (PRMs) face structural limitations due to their reliance on scalar value predictions without leveraging the generative capabilities of language models, which restricts their process supervision, generalization capabilities, and ability to leverage Test-Time Compute Scaling. Step-level generative verification overcomes these constraints by requiring the model to perform explicit Chain-of-Thought (CoT) reasoning alongside code verification before rendering a judgment for each individual reasoning step.

    In frameworks such as GenPRM, step-level generative verification utilizes Relative Progress Estimation (RPE) and a rationale synthesis framework that integrates code verification to produce high-quality process supervision labels and explicit rationale data. This structured step-level rationale generation allows process verification to unlock Test-Time Compute Scaling. Consequently, a 1.5B GenPRM model leverages Test-Time Compute Scaling to outperform GPT-4o, while a 7B GenPRM model surpasses Qwen2.5-Math-PRM-72B on ProcessBench using only 23K training instances from the MATH dataset. Beyond basic step evaluation, generating explicit reasoning and code verification for each step enables the verifier to serve as a critic model for policy model refinement, bridging the operational gap between PRMs and critic models.

    Sources: GenPRM: Scaling Test-Time Compute of Process Reward Models via Generative Reasoning

    Execution-Based Reward Models for Code Generation Search

    While generative verification strengthens step-level scoring, code generation presents a related but distinct challenge for search-based inference. Rather than scoring intermediate reasoning steps directly like a classic PRM, search and verification mechanisms in this domain frequently generate multiple complete candidate solutions and validate each one using execution results from model-generated unit tests as an outcome-based reward signal. However, because models can produce flawed tests with high confidence, these test-based reward signals are often unreliable, degrading the overall quality of candidate verification.

    Scaling the number of unit tests provides a positive correlation with reward signal quality, yielding higher performance benefits on more challenging problems. To balance reward precision with execution efficiency during search, a dynamic scaling mechanism can adaptively adjust the quantity of generated unit tests according to problem difficulty. Utilizing CodeRM-8B—a lightweight unit test generator designed for high-quality scaling—alongside dynamic unit test scaling delivers marked accuracy improvements across multiple benchmarks, including performance gains of 18.43% for Llama3-8B and 3.42% for GPT-4o-mini on HumanEval Plus.

    Sources: Dynamic Scaling of Unit Tests for Code Reward Modeling

    Latency Overhead and Operational Cost Consequences

    Although integrating search and verification mechanisms significantly boosts accuracy, scaling compute during inference imposes severe latency overhead and high operational inference costs during active user interactions. To mitigate these execution-time constraints, offline pre-computation approaches such as Sleep-time Compute can reallocate compute demands by allowing models to “think” offline about context prior to query presentation. By anticipating user queries and pre-computing useful quantities ahead of time, this approach significantly decreases the compute required during the actual test-time phase.

    Empirical evaluations on Stateful GSM-Symbolic and Stateful AIME show that Sleep-time Compute reduces the amount of Test-Time Compute Scaling needed to achieve equivalent accuracy by approximately 5x. Furthermore, scaling the amount of Sleep-time Compute allocated offline yields direct accuracy improvements, driving gains of up to 13% on Stateful GSM-Symbolic and 18% on Stateful AIME.

    To further lower per-query operational costs, Sleep-time Compute can be amortized across multiple requests sharing the same context. In Multi-Query GSM-Symbolic—an extension incorporating multiple related queries per context—amortizing pre-computed quantities across these queries reduces the average compute cost per query by 2.5x. Analysis indicates that the predictability of incoming user queries correlates directly with the effectiveness of Sleep-time Compute in reducing test-time latency and operational overhead. Case study evidence confirms the applicability of this strategy to complex, realistic workloads, including agentic software engineering (SWE) tasks.

    Sources: Sleep-time Compute: Beyond Inference Scaling at Test-time

    Closing thoughts

    Ultimately, the evidence demonstrates that shifting from coarse outcome evaluations to step-level generative verification allows compact models to achieve superior reasoning by systematically catching intermediate errors at their point of origin. However, because scaling search and verification during live inference introduces severe latency and operational cost bottlenecks, relying solely on real-time verifications remains practically constrained. By pairing dynamic step-level feedback with offline strategies like Sleep-time Compute, systems can pre-calculate context and amortize overhead across predictable queries to drastically reduce test-time costs. In my view, the true promise of test-time scaling lies in this precise synergy—balancing fine-grained process supervision with intelligent offline compute allocation to deliver high-accuracy reasoning efficiently.

    Frequently Asked Questions

    What is the main difference between Outcome Reward Models (ORMs) and Process Reward Models (PRMs)?

    Outcome Reward Models evaluate only the final output of a completed response, which can result in rewarding flawed intermediate logic. In contrast, Process Reward Models evaluate each individual intermediate step in a Chain-of-Thought trajectory to catch reasoning errors at their point of origin.

    How does step-level generative verification improve reasoning over traditional PRMs?

    Traditional PRMs rely on scalar value predictions without using the generative capabilities of language models. Step-level generative verification overcomes this limitation by requiring explicit Chain-of-Thought reasoning alongside code verification before judging each step.

    What are two main architectural approaches to using step-level feedback during inference?

    The two main approaches are External Search with PRMs, which uses dedicated verifiers to steer explicit search algorithms, and Implicit Test-Time Compute Scaling via RL, which trains policy models to generate self-correcting internal reasoning tokens in a single pass.

    How does Sleep-time Compute help mitigate test-time latency overhead?

    Sleep-time Compute reallocates compute demands by allowing models to pre-compute context offline before a user presents a query. This reduces the test-time compute required to achieve equivalent accuracy by approximately 5x and lowers per-query costs when amortized across multiple requests.

  • Kolmogorov-Arnold Networks: Rethinking Neural Architecture via Learnable Spline Activations

    Kolmogorov-Arnold Networks: Rethinking Neural Architecture via Learnable Spline Activations

    TL;DR

    • Kolmogorov-Arnold Networks (KANs) serve as an alternative to Multi-Layer Perceptrons by placing learnable B-spline activation functions directly on network edges to model complex, non-monotonic dependencies.
    • KANs significantly enhance model interpretability in applications such as cognitive diagnosis while achieving competitive training efficiency through optimized implementations.
    • Architectural extensions like the Kurkova-Kolmogorov-Arnold Network outperform traditional MLPs and original KANs in function approximation and operator learning tasks.

    Introduction to Kolmogorov-Arnold Networks (KANs)

    Kolmogorov-Arnold Networks (KANs) were introduced in April 2024 by a team of researchers from MIT, Caltech, and Northeastern University (several affiliated with the NSF Institute for Artificial Intelligence and Fundamental Interactions): Ziming Liu, Yixuan Wang, Sachin Vaidya, Fabian Ruehle, James Halverson, Marin Soljačić, Thomas Y. Hou, and Max Tegmark. Accepted as an oral presentation at ICLR 2025, KANs offer a fundamentally different neural network architecture intended to serve as an alternative to standard Multi-Layer Perceptrons (MLPs).

    Sources: arxiv.org, iclr.cc, arxiv.org, arxiv.org, github.com

    Mathematical Foundation and Structural Differences from MLPs

    At the core of this architectural alternative is the mathematical foundation provided by the Kolmogorov-Arnold Representation Theorem. One notable application of this foundation outside standard neural networks is in Fuzzy Cognitive Maps (FCMs), a graph-based reasoning framework where nodes represent factors and edges represent causal influence between them. Drawing upon this theorem, Kolmogorov-Arnold Fuzzy Cognitive Maps (KA-FCMs) redefine the causal transmission mechanism to overcome the limitations of standard FCMs. The standard FCM formulation relies on static scalar synaptic weights and monotonic activation functions, placing non-linearity at the nodes’ aggregation phase, which fundamentally constrains its ability to model non-monotonic causal dependencies. In contrast, the KA-FCM architecture replaces static scalar weights with learnable univariate B-spline activation functions located directly on the model edges. This shift moves the non-linearity from the node aggregation phase directly to the causal influence phase along the edges, enabling the modeling of arbitrary, non-monotonic causal relationships without increasing graph density or introducing hidden layers.

    Sources: Non-monotonic causal discovery with Kolmogorov-Arnold Fuzzy Cognitive Maps

    Model Interpretability and Efficiency Benefits

    Beyond these core mathematical properties, this shift in architecture provides significant advantages in model interpretability and efficiency. In domain-specific applications such as intelligent education and cognitive diagnosis, Kolmogorov-Arnold Networks (KANs) address the long-standing interpretability limitations of Multi-Layer Perceptrons (MLPs). Neural Cognitive Diagnosis Models rely on embeddings for students, exercises, and knowledge concepts to reveal proficiency for downstream recommendation tasks. While neural Cognitive Diagnosis Models typically outperform traditional models, their reliance on MLPs yields poor interpretability even when constrained by monotonicity assumptions. Replacing MLPs with KANs in cognitive diagnosis architectures—such as in the KAN2CD framework—enhances model interpretability through two structural designs: direct replacement of standard MLPs in existing neural Cognitive Diagnosis Models with KANs, and hierarchical combination of student, exercise, and concept embeddings through several distinct KANs before combining and learning their outputs within a unified KAN to generate final predictions. The learned structures of KANs allow these enhanced neural Cognitive Diagnosis Models to maintain the clear interpretability of traditional Cognitive Diagnosis Models while surpassing existing neural Cognitive Diagnosis Models in interpretability.

    Regarding computational efficiency, original KAN architectures present a challenge due to slow training speeds. However, modifying the implementation of original KANs accelerates training, yielding computational costs that are competitive with existing models. On four real-world datasets, efficient KAN-based models outperform traditional Cognitive Diagnosis Models and maintain a performance lead over existing neural Cognitive Diagnosis Models while retaining high interpretability and competitive training costs.

    Sources: Endowing Interpretability for Neural Cognitive Diagnosis by Efficient Kolmogorov-Arnold Networks

    Architectural Extensions and Learning Dynamics

    Building upon these interpretability and efficiency benefits, researchers have also introduced broader architectural extensions and examined their learning dynamics. Inspired by the Kolmogorov-Arnold Representation Theorem and Kurkova’s principle of using approximate representations, the Kurkova-Kolmogorov-Arnold Network (KKAN) is a two-block architecture that combines robust Multi-Layer Perceptron (MLP) based inner functions with flexible linear combinations of basis functions as outer functions. Proven to be a universal approximator, KKAN demonstrates versatility across scientific machine-learning applications, including function regression, Physics-Informed Machine Learning (PIML), and operator-learning frameworks. In benchmark results, KKANs outperform MLPs and original Kolmogorov-Arnold Networks (KANs) in function approximation and operator learning tasks, while achieving performance comparable to fully optimized MLPs for PIML.

    An analysis using information bottleneck theory provides insight into the geometric complexity and learning dynamics of these models, identifying three universal learning stages across all types of architectures: fitting, transition, and diffusion. Optimal generalization is achieved during the diffusion stage. Additionally, a strong correlation exists between geometric complexity and signal-to-noise ratio (SNR). To dynamically maintain a high SNR, self-scaled residual-based attention weights can be used, ensuring uniform convergence and prolonged learning.

    Sources: KKANs: Kurkova-Kolmogorov-Arnold Networks and Their Learning Dynamics

    Closing thoughts

    Taking these empirical and structural insights together, Kolmogorov-Arnold Networks demonstrate how relocating non-linear activation functions directly onto network edges via learnable B-splines provides a compelling alternative to traditional MLPs that fundamentally enhances how complex, non-monotonic dependencies are modeled. In my view, this paradigm shift successfully resolves the historical trade-off between interpretability and predictive power, as evidenced by clear explainability gains in cognitive diagnosis tasks and superior function approximation in hybrid extensions like KKANs. As optimized implementations continue to resolve early training inefficiencies and leverage dynamic controls like residual-based attention, KAN-based architectures demonstrate that rethinking foundational neural building blocks can yield significant gains in transparency without sacrificing performance or computational viability.

    Frequently Asked Questions

    What are Kolmogorov-Arnold Networks (KANs) and who introduced them?

    Kolmogorov-Arnold Networks (KANs) are an alternative neural network architecture to standard Multi-Layer Perceptrons (MLPs). They were introduced in April 2024 by a team of researchers from MIT, Caltech, and Northeastern University.

    How do KANs differ structurally from standard Multi-Layer Perceptrons?

    KANs replace static scalar weights with learnable univariate B-spline activation functions located directly on the model edges. This moves non-linearity from the node aggregation phase to the edges, enabling the model to represent arbitrary non-monotonic causal relationships.

    How do KANs address interpretability and efficiency challenges?

    KANs improve interpretability in domain-specific applications like cognitive diagnosis, matching the explainability of traditional models while maintaining neural performance. Additionally, modified KAN implementations accelerate training speeds to make computational costs competitive with existing models.

    What is the Kurkova-Kolmogorov-Arnold Network (KKAN)?

    The KKAN is a two-block architectural extension that combines MLP-based inner functions with flexible linear combinations of basis functions as outer functions. Proven to be a universal approximator, it outperforms standard MLPs and original KANs in function approximation and operator learning tasks.

  • Demystifying Mixture of Experts Routing: Core Mechanisms, Optimization, and Practical Challenges

    Demystifying Mixture of Experts Routing: Core Mechanisms, Optimization, and Practical Challenges

    TL;DR

    • Mixture of Experts architectures scale model capacity while keeping active computational costs low by using a gating network to route tokens to specific expert subnetworks.
    • Empirical analysis demonstrates that expert activation patterns are highly task-conditioned, with prompts from the same category producing highly similar routing signatures.
    • To resolve single-GPU memory bottlenecks, deployment strategies utilize post-training expert pruning and predictive caching systems like ExpertFlow to reduce memory usage and improve throughput.

    An Introduction to Mixture of Experts (MoE) Routing

    MoE routing represents a core architectural component in modern large language models (LLMs), serving as the foundational mechanism for conditional computation. This approach allows a model to scale its overall parameter capacity to hundreds of billions or even trillions of parameters. Crucially, it achieves this scaling while keeping the active computational cost—the number of parameters actually activated to process a given input—equivalent to that of a much smaller model.

    In a standard Transformer architecture, every input token in a sequence is processed by the exact same Feed-Forward Network (FFN) at each layer. A SparseMoE model restructures this setup by replacing that single FFN with multiple parallel subnetworks known as “experts.” To coordinate this architecture, a gating network is positioned before the expert layer. The gating network is a lightweight, trainable neural network that functions as a traffic director. For each incoming token, it determines which specific experts are most mathematically suited to handle the processing and directs the token to them accordingly.

    The standard mathematical execution of MoE routing operates through three primary steps:

    1. Compute Logits: For an incoming token representation x, the gating network computes a dot product with a learnable weight matrix W_g to produce affinity scores (logits) for each expert:
      Logits = x * W_g

    2. Top-k Routing: Rather than sending the token to all experts, which would be computationally expensive, the gating network selects only the top-k experts that yielded the highest affinity scores. In standard setups, k is typically set to 1 or 2.

    3. Softmax Normalization and Weighted Combine: The logits of the selected experts are normalized using a softmax function. Once these chosen experts process the token, their respective outputs are multiplied by these normalized gating network probabilities and summed together to produce the final representation of the token.

    Sources: huggingface.co, research.google, maartengrootendorst.com, thelmbook.com, tistory.com

    Routing Paradigms: Top-k vs. Soft Routing Mechanisms

    Building on these standard routing mechanics, alternative paradigms have emerged to optimize performance and stability. When evaluating SoftMoE and SparseMoE classifier heads under comparable model capacity, both routing paradigms achieve slightly higher validation accuracy than a dense baseline on the CIFAR10 image classification dataset. To avoid routing collapse, both variants prevent load imbalance through auxiliary loss.

    The primary distinctions between these paradigms emerge in their generalization and loss surface curvature. Hessian-based sharpness metrics at convergence—specifically the largest eigenvalue and the trace of the loss Hessian evaluated on both training and test data—show that SoftMoE exhibits higher sharpness. Meanwhile, SparseMoE and dense models lie in a similar curvature regime, despite all of these models achieving comparable generalization performance. Furthermore, loss surface perturbation analyses show qualitative differences in non-local behavior under finite parameter perturbations between dense and Mixture of Experts models.

    In terms of operational performance, there is a pronounced gap between theoretical and realized efficiency in SparseMoE models. Naively implemented MoE routing fails to yield empirical inference speedups on modern hardware at this scale, highlighting the practical challenges of deploying sparse routing mechanisms.

    Sources: Mixture-of-Experts Models in Vision: Routing, Optimization, and Generalization

    Analyzing Real-World Routing: Task-Conditioned Signatures

    To address these practical deployment challenges and gain deeper insights into how experts actually behave, researchers have introduced routing signatures as a tool to analyze whether expert selection in Mixture of Experts architectures exhibits task-conditioned structure. A routing signature is a vector representation that summarizes expert activation patterns across layers for a given prompt.

    Empirical analysis using the OLMoE-1B-7B-0125-Instruct model as a testbed demonstrates that prompts from the same task category induce highly similar routing signatures, whereas prompts from different categories have significantly lower similarity. Specifically, within-category routing similarity reaches 0.8435 +/- 0.0879, which substantially exceeds the across-category similarity of 0.6225 +/- 0.1687, representing a Cohen’s d of 1.44. The distinctiveness of these signatures is strong enough that a simple logistic regression classifier trained solely on routing signatures can achieve 92.5% +/- 6.1% cross-validated accuracy on four-way task classification.

    To confirm that this separation is a genuine reflection of task-specific routing rather than an artifact of architectural constraints, the analysis incorporated permutation and load imbalance baselines. These baselines demonstrate that the observed task-based clustering cannot be explained by sparsity or load imbalance constraints alone. Furthermore, examining layer-wise signal strength and low-dimensional projections of routing signatures reveals that this task-conditioned structure becomes increasingly apparent in the deeper layers of the model. These findings indicate that routing in sparse transformers operates as a measurable, task-sensitive component of conditional computation rather than serving merely as a load imbalance prevention mechanism. To facilitate further routing telemetry and analysis, researchers have released a lightweight toolkit named MOE-XRAY.

    Sources: Task-Conditioned Routing Signatures in Sparse Mixture-of-Experts Transformers

    Addressing Memory Bottlenecks: Expert Pruning and Predictive Caching

    Beyond analyzing routing signatures, another major hurdle in utilizing these architectures is managing their immense parameter sizes. To mitigate this heavy memory footprint, particularly in memory-constrained environments like single-GPU deployments, researchers have developed post-training sparsification techniques and predictive offloading systems.

    One primary direction to optimize deployment efficiency is pruning and skipping inactive or redundant components. Recent research has introduced post-training approaches that provide plug-and-play, expert-level sparsification through task-agnostic and task-specific expert pruning and expert skipping. Unlike historical weight pruning methods that rely on specialized hardware, these techniques prune or skip entire experts to simultaneously reduce overall model sizes and increase inference speeds while maintaining satisfactory performance across a wide range of tasks.

    Another approach to overcoming memory bottlenecks on a single GPU is dynamic offloading, where inactive experts are stored in CPU memory and loaded on demand. Because static caches disregard input-dependent routing and training separate predictor models is often inaccurate or computationally expensive, the ExpertFlow inference system coordinates three distinct components to manage routing dependencies efficiently:

    • Transformer-based routing path predictor: This component estimates expert usage across all Mixture of Experts layers in a single forward pass.
    • Token scheduler: This groups tokens with similar predicted routes to maximize expert utilization.
    • Predictive expert cache: This system loads only the necessary experts into memory while dynamically correcting any routing mispredictions at runtime.

    By coordinating these three components, ExpertFlow enables highly efficient expert loading and execution. On a single GPU, this predictive caching and scheduling pipeline reduces GPU memory usage by up to 93.72% and improves inference throughput by up to 10x compared to strong offloading baselines.

    Sources: ExpertFlow: Efficient Mixture-of-Experts Inference via Predictive Expert Caching and Token Scheduling, Not All Experts are Equal: Efficient Expert Pruning and Skipping for Mixture-of-Experts Large Language Models

    Closing thoughts

    Ultimately, as these optimization techniques demonstrate, the viability of Mixture of Experts architectures hinges on closing the pronounced gap between their theoretical computational elegance and the physical realities of modern hardware deployment. While naive routing struggles with empirical inference speeds, the discovery of highly structured, task-conditioned routing signatures in deeper layers demonstrates that these gating networks are executing genuine, meaningful computation rather than arbitrary load balancing. Consequently, by leveraging this predictable behavior through targeted expert pruning and dynamic predictive caching systems like ExpertFlow, developers can successfully resolve severe single-GPU memory bottlenecks. This suggests that the true maturity of MoE technology will be defined not merely by expanding model sizes, but by co-designing hardware-aware routing algorithms with robust diagnostic telemetry.

    Frequently Asked Questions

    What is a routing signature in Mixture of Experts (MoE) architectures?

    A routing signature is a vector representation that summarizes expert activation patterns across layers for a given prompt. It is used by researchers to analyze whether expert selection in MoE models exhibits task-conditioned structures.

    How does the ExpertFlow system optimize single-GPU memory and inference performance?

    ExpertFlow coordinates a Transformer-based routing path predictor, a token scheduler, and a predictive expert cache to manage routing dependencies. This pipeline reduces GPU memory usage by up to 93.72% and improves inference throughput by up to 10x compared to strong offloading baselines.

    What are the three steps involved in standard MoE routing?

    First, the gating network computes affinity scores (logits) for each expert using a dot product. Second, top-k routing selects only the experts with the highest scores, typically setting k to 1 or 2. Finally, the chosen experts’ outputs are normalized using a softmax function and summed together to produce the final representation.