Kolmogorov-Arnold Networks: Rethinking Neural Architecture via Learnable Spline Activations

Introduction to Kolmogorov-Arnold Networks (KANs)

Kolmogorov-Arnold Networks (KANs) were introduced in April 2024 by a team of researchers from MIT, Caltech, and Northeastern University (several affiliated with the NSF Institute for Artificial Intelligence and Fundamental Interactions): Ziming Liu, Yixuan Wang, Sachin Vaidya, Fabian Ruehle, James Halverson, Marin Soljačić, Thomas Y. Hou, and Max Tegmark. Accepted as an oral presentation at ICLR 2025, KANs offer a fundamentally different neural network architecture intended to serve as an alternative to standard Multi-Layer Perceptrons (MLPs).

Sources: arxiv.org, iclr.cc, arxiv.org, arxiv.org, github.com

Mathematical Foundation and Structural Differences from MLPs

At the core of this architectural alternative is the mathematical foundation provided by the Kolmogorov-Arnold Representation Theorem. One notable application of this foundation outside standard neural networks is in Fuzzy Cognitive Maps (FCMs), a graph-based reasoning framework where nodes represent factors and edges represent causal influence between them. Drawing upon this theorem, Kolmogorov-Arnold Fuzzy Cognitive Maps (KA-FCMs) redefine the causal transmission mechanism to overcome the limitations of standard FCMs. The standard FCM formulation relies on static scalar synaptic weights and monotonic activation functions, placing non-linearity at the nodes’ aggregation phase, which fundamentally constrains its ability to model non-monotonic causal dependencies. In contrast, the KA-FCM architecture replaces static scalar weights with learnable univariate B-spline activation functions located directly on the model edges. This shift moves the non-linearity from the node aggregation phase directly to the causal influence phase along the edges, enabling the modeling of arbitrary, non-monotonic causal relationships without increasing graph density or introducing hidden layers.

Sources: Non-monotonic causal discovery with Kolmogorov-Arnold Fuzzy Cognitive Maps

Model Interpretability and Efficiency Benefits

Beyond these core mathematical properties, this shift in architecture provides significant advantages in model interpretability and efficiency. In domain-specific applications such as intelligent education and cognitive diagnosis, Kolmogorov-Arnold Networks (KANs) address the long-standing interpretability limitations of Multi-Layer Perceptrons (MLPs). Neural Cognitive Diagnosis Models rely on embeddings for students, exercises, and knowledge concepts to reveal proficiency for downstream recommendation tasks. While neural Cognitive Diagnosis Models typically outperform traditional models, their reliance on MLPs yields poor interpretability even when constrained by monotonicity assumptions. Replacing MLPs with KANs in cognitive diagnosis architectures—such as in the KAN2CD framework—enhances model interpretability through two structural designs: direct replacement of standard MLPs in existing neural Cognitive Diagnosis Models with KANs, and hierarchical combination of student, exercise, and concept embeddings through several distinct KANs before combining and learning their outputs within a unified KAN to generate final predictions. The learned structures of KANs allow these enhanced neural Cognitive Diagnosis Models to maintain the clear interpretability of traditional Cognitive Diagnosis Models while surpassing existing neural Cognitive Diagnosis Models in interpretability.

Regarding computational efficiency, original KAN architectures present a challenge due to slow training speeds. However, modifying the implementation of original KANs accelerates training, yielding computational costs that are competitive with existing models. On four real-world datasets, efficient KAN-based models outperform traditional Cognitive Diagnosis Models and maintain a performance lead over existing neural Cognitive Diagnosis Models while retaining high interpretability and competitive training costs.

Sources: Endowing Interpretability for Neural Cognitive Diagnosis by Efficient Kolmogorov-Arnold Networks

Architectural Extensions and Learning Dynamics

Building upon these interpretability and efficiency benefits, researchers have also introduced broader architectural extensions and examined their learning dynamics. Inspired by the Kolmogorov-Arnold Representation Theorem and Kurkova’s principle of using approximate representations, the Kurkova-Kolmogorov-Arnold Network (KKAN) is a two-block architecture that combines robust Multi-Layer Perceptron (MLP) based inner functions with flexible linear combinations of basis functions as outer functions. Proven to be a universal approximator, KKAN demonstrates versatility across scientific machine-learning applications, including function regression, Physics-Informed Machine Learning (PIML), and operator-learning frameworks. In benchmark results, KKANs outperform MLPs and original Kolmogorov-Arnold Networks (KANs) in function approximation and operator learning tasks, while achieving performance comparable to fully optimized MLPs for PIML.

An analysis using information bottleneck theory provides insight into the geometric complexity and learning dynamics of these models, identifying three universal learning stages across all types of architectures: fitting, transition, and diffusion. Optimal generalization is achieved during the diffusion stage. Additionally, a strong correlation exists between geometric complexity and signal-to-noise ratio (SNR). To dynamically maintain a high SNR, self-scaled residual-based attention weights can be used, ensuring uniform convergence and prolonged learning.

Sources: KKANs: Kurkova-Kolmogorov-Arnold Networks and Their Learning Dynamics

Closing thoughts

Taking these empirical and structural insights together, Kolmogorov-Arnold Networks demonstrate how relocating non-linear activation functions directly onto network edges via learnable B-splines provides a compelling alternative to traditional MLPs that fundamentally enhances how complex, non-monotonic dependencies are modeled. In my view, this paradigm shift successfully resolves the historical trade-off between interpretability and predictive power, as evidenced by clear explainability gains in cognitive diagnosis tasks and superior function approximation in hybrid extensions like KKANs. As optimized implementations continue to resolve early training inefficiencies and leverage dynamic controls like residual-based attention, KAN-based architectures demonstrate that rethinking foundational neural building blocks can yield significant gains in transparency without sacrificing performance or computational viability.

Frequently Asked Questions

What are Kolmogorov-Arnold Networks (KANs) and who introduced them?

Kolmogorov-Arnold Networks (KANs) are an alternative neural network architecture to standard Multi-Layer Perceptrons (MLPs). They were introduced in April 2024 by a team of researchers from MIT, Caltech, and Northeastern University.

How do KANs differ structurally from standard Multi-Layer Perceptrons?

KANs replace static scalar weights with learnable univariate B-spline activation functions located directly on the model edges. This moves non-linearity from the node aggregation phase to the edges, enabling the model to represent arbitrary non-monotonic causal relationships.

How do KANs address interpretability and efficiency challenges?

KANs improve interpretability in domain-specific applications like cognitive diagnosis, matching the explainability of traditional models while maintaining neural performance. Additionally, modified KAN implementations accelerate training speeds to make computational costs competitive with existing models.

What is the Kurkova-Kolmogorov-Arnold Network (KKAN)?

The KKAN is a two-block architectural extension that combines MLP-based inner functions with flexible linear combinations of basis functions as outer functions. Proven to be a universal approximator, it outperforms standard MLPs and original KANs in function approximation and operator learning tasks.