Blog
-
Fixing Quality Degradation When Merging 4-Bit QLoRA Adapters in PEFT
TL;DR In QLoRA, the base model weights are stored in 4-bit precision (NormalFloat4 / NF4 via bitsandbytes) while the trained LoRA adapter matrices stay in 16-bit floating point (torch.float16 or torch.bfloat16). Calling merge_and_unload directly on a model that is still loaded in 4-bit forces PEFT…
-
Byte Latent Transformer: Replacing Tokenizers with Dynamic Entropy-Based Patching
TL;DR The Byte Latent Transformer introduces a tokenizer-free architecture that overcomes the noise brittleness and vocabulary biases of traditional subword tokenization. It processes raw byte streams efficiently by dynamically segmenting bytes into patches based on next-byte entropy. BLT matches or exceeds the performance of traditional…
-
Resolving Pydantic State Deserialization Failures in LangGraph Checkpointers
TL;DR Pydantic state deserialization problems in LangGraph checkpointers come from four distinct mechanisms with different symptoms: msgpack allowlist restrictions under strict mode, classes that cannot be re-imported (dynamic or local scope), schema drift between old checkpoints and newer models, and non-serializable attributes that actually fail…
-
Titans: Learning to Memorize at Test Time with Neural Long-Term Memory
TL;DR Titans introduces a dual-memory architecture combining attention for short-term dependencies with a Neural Long-Term Memory module for persistent historical context. The framework resolves trade-offs between quadratic attention and lossy recurrent models through three variants: Memory as Context, Memory as Gate, and Memory as Layer.…
-
Resolving SQLite Database Lock Errors in ChromaDB During Concurrent Writes
TL;DR SQLite enforces a global write lock in Embedded ChromaDB, which causes SQLITE_BUSY database lock errors during concurrent write operations and read-to-write lock upgrades. Standard write contention can be mitigated by keeping insert batch sizes between 50 and 250 records and raising SQLite’s own PRAGMA…
-
Group Relative Policy Optimization for Efficient Reinforcement Learning in Language Models
TL;DR Group Relative Policy Optimization (GRPO) eliminates the Critic Model from standard PPO architectures, cutting GPU VRAM usage nearly in half during LLM reinforcement learning. Standard GRPO relies on static sampling and fixed rollouts, which can waste computational resources on easy prompts while under-training difficult…
-
Fixing Cannot Copy Out of Meta Tensor Errors in Hugging Face Transformers
TL;DR The ‘Cannot copy out of meta tensor’ error happens when attempting manual operations like .to(device) on meta tensors, which serve as memory-free model placeholders. Avoid calling .to() manually when loading models with device_map=’auto’, as Hugging Face Accelerate automatically handles layer placement and device transfers…
-
SimPO: Reference-Free Preference Optimization for Large Language Models
TL;DR Developed by researchers at Princeton and UVA, SimPO is a reference-free preference optimization method that eliminates the need for a frozen reference model in LLM alignment. By utilizing sequence-average log probabilities and a target reward margin, SimPO aligns training objectives with inference while reducing…
-
Fixing High CPU Spikes and Freezes During Large Codebase Indexing in Cursor
TL;DR High CPU spikes during Cursor codebase indexing are usually not an embedding-generation cost — they mostly come from background rg (ripgrep) processes scanning large binary/artifact-heavy workspaces with wide-scope flags. Developers can shrink that scan scope by excluding heavy files with .cursorignore, .cursorindexingignore, and a…
-
Fixing Silent PyTorch DataLoader Crashes in Docker Containers
TL;DR PyTorch DataLoader uses POSIX shared memory at /dev/shm to efficiently pass CPU tensor data between parallel worker processes. Docker containers allocate a default limit of only 64 MB to /dev/shm, causing workers to crash with a SIGBUS signal once that space is exhausted —…