Every language model you’ve ever used — GPT-4, Claude, Gemini, LLaMA — starts with the same preprocessing step before a single parameter is touched.
A tokenizer runs over your input and converts it into a sequence of integer IDs. "Hello world" becomes something like [9906, 1917]. The model never sees characters. It sees token IDs drawn from a fixed vocabulary, usually 32K to 128K entries, built offline by running BPE or SentencePiece on a large corpus.
This has been the unquestioned default for five years.
In late 2024, Meta AI published “Byte Latent Transformer: Patches Scale Better Than Tokens” — and questioned it.
The core claim: you can train a competitive language model directly on raw bytes, no tokenizer required, and at the same compute budget it matches or beats LLaMA 3 on most benchmarks. On tasks that expose tokenization’s weaknesses — character manipulation, rare languages, structured data — it wins clearly.
This post covers why tokenization has always been a compromise, how BLT replaces it, and why this architecture matters.
Before explaining what BLT does, it’s worth being specific about what tokenizers get wrong.
BPE (Byte-Pair Encoding) builds a vocabulary by iteratively merging the most frequent character pairs in a training corpus. The result is deeply English-centric. Common English words get single tokens. Rare words, names, and most non-Latin scripts get shredded into fragments.
The word "unfortunately" → 1 token. The Turkish word "değiştiremeyebilirsiniz" → 12+ tokens.
Same information density, wildly different token cost. Multilingual models pay a massive efficiency penalty on non-English text simply because of vocabulary construction choices made before training started.
Because models operate on token IDs rather than characters, they have no direct access to spelling. This causes failures that look baffling from the outside:
9.11 > 9.9 — GPT-4 originally said false. The numbers are tokenized as whole units; the model sees 9 and .11 vs 9 and .9 without clean positional structure.r in "strawberry" → famously hard for tokenized models, trivial for humans.These aren’t reasoning failures. They’re tokenization artifacts.
Adding a space changes the token. Capitalization changes the token. A URL with an unusual structure might produce a token sequence no model has seen during training. Code with uncommon variable names gets fragmented unpredictably.
Models can recover from this because they have enough capacity to memorize patterns — but they’re burning parameters on compensating for preprocessing noise.
If a new domain emerges after the tokenizer is built — a new programming language, a new file format, a new script — the model handles it poorly. The vocabulary is frozen. There’s no mechanism to learn better representations for new patterns without retraining the tokenizer and rebuilding the vocabulary from scratch.
At the other extreme: raw bytes.
Every piece of digital information — text in any language, code, HTML, JSON, images, audio — can be represented as a sequence of bytes. A byte is a number from 0 to 255. There are exactly 256 possible values.
A byte-level model has a vocabulary of 256. It never needs to be retrained. It never has an OOV (out-of-vocabulary) problem. Every string in any language on any format is representable without special handling.
The catch: raw bytes produce very long sequences. "Hello world" is 11 bytes. A 10,000-word document is ~60,000 bytes. Transformers scale quadratically with sequence length — so naive byte-level transformers are expensive, slow, and don’t scale well.
This is why the field defaulted to tokenization in the first place. Tokens compress the sequence. Shorter sequence → cheaper attention → more compute-efficient training.
BLT’s contribution is solving the byte-length problem without reintroducing a fixed vocabulary.
BLT replaces the tokenizer with a learned, dynamic patching mechanism. Instead of a fixed vocabulary, it groups bytes into variable-length patches based on how predictable the bytes are.
The full model has three components:
Raw bytes
↓
Local Encoder (lightweight byte-level transformer)
↓
Patch representations (dynamic, variable-length groups)
↓
Latent Transformer (large, expensive — operates on patches)
↓
Patch predictions
↓
Local Decoder (reconstructs byte-level output from patches)
↓
Output bytes
A small, lightweight transformer that reads raw bytes and produces byte-level representations. It doesn’t try to understand the whole document — it just builds local context around each byte.
The output feeds into the patching step.
This is the core idea. Instead of splitting text at fixed boundaries (like BPE does), BLT groups bytes into patches based on next-byte entropy — a measure of how surprising the next byte is.
High entropy = the model is uncertain what comes next. This byte starts something complex. Start a new patch here.
Low entropy = the next byte is predictable. This byte is part of a familiar pattern. Extend the current patch.
Entropy-based patching example:
Input: " t h e q u i c k b r o w n f o x "
Entropy: [H L L H L L L L H L L L L H L L L]
Patches: [_the] [_quick] [_brown] [_fox]
Common words and phrases get compressed into single patches. Rare words, code snippets, unusual names, and multilingual text get finer-grained patches — more attention per byte, exactly where it’s needed.
This is the key insight: allocate compute proportional to information content. Predictable bytes get bundled cheaply. Surprising bytes get resolved carefully.
The patch boundaries are dynamic — they change based on the actual input. A technical paper about nuclear physics gets different patches than a children’s story, even at the same byte length.
The main, expensive transformer. It operates on patch representations, not individual bytes. Because patches compress the sequence — typically by a factor of 4–8× compared to raw bytes — the latent transformer sees much shorter sequences.
This is where the bulk of the model’s parameters live and where the “understanding” happens. It uses standard transformer architecture — multi-head attention, feed-forward layers — but on patch-level units.
Critically, patch representations carry context from the Local Encoder via cross-attention. The latent transformer can look back at individual byte-level details when it needs to.
Reconstructs byte-level output from the patch-level predictions. Given a predicted patch, it generates the actual bytes that form the output — one byte at a time, conditioned on both the patch prediction and previously generated bytes.
The decoder is also lightweight. Most of the compute stays in the Latent Transformer.
The paper’s title makes a specific claim: patches scale better than tokens.
Here’s the intuition.
As models get larger, they get better at predicting common patterns — frequent words, standard syntax, typical phrases. With fixed tokenization, every token has the same representation cost regardless of how predictable it is. A token for "the" takes the same slot in the attention matrix as a token for "Schrödinger".
With entropy-based patching, as the model improves:
This creates a virtuous loop that fixed tokenization can’t replicate. Tokenization is static — it doesn’t adapt to what the model has learned. BLT’s patching is dynamic — it improves with the model.
The paper shows this empirically: at the same compute budget, BLT matches LLaMA 3. At larger scales, the efficiency gap favors BLT more.
Some tasks are structurally impossible to do well with tokenized models. BLT handles all of them naturally.
Reverse a string, count specific characters, detect palindromes, anagram matching. BLT operates at the byte level — these tasks are as natural for it as scanning a character array is for you.
No vocabulary bias. Turkish, Arabic, Hindi, Chinese, code-switching mid-sentence — all represented at the same byte-level granularity. No 10× token penalty for non-Latin scripts.
JSON, CSV, XML, HTML, source code with unusual variable names — all handled without tokenization artifacts. The entropy-based patcher naturally segments these formats at structurally meaningful boundaries.
A tokenizer turns a misspelled word into a completely different token sequence. BLT sees character-level corruption as a byte-level signal and can reason about it directly. This matters for real-world noisy input.
A new programming language, a new markup format, a new domain — BLT handles it without retraining the vocabulary. There’s no concept of OOV. The 256-byte vocabulary is universal.
| Benchmark | LLaMA 3 (tokens) | BLT (bytes) |
|---|---|---|
| FLOP budget | Matched | Matched |
| Overall performance | Baseline | Matches or beats |
| Character tasks | Weak | Significantly better |
| Multilingual | Penalized by vocab | Equal cost per byte |
| Robustness | Token-brittle | Byte-robust |
| Vocabulary size | 128K | 256 (universal) |
| OOV handling | Fragmentation | None |
At equal compute: BLT is competitive. At tasks that expose tokenization’s weaknesses: BLT wins clearly.
BLT is a compelling paper but not a solved problem.
Inference speed is still a challenge. The Local Encoder and Decoder add overhead at inference time that pure token-based models don’t have. The latent transformer may be cheaper due to shorter sequences, but the total wall-clock cost per output byte needs careful engineering.
The compression factor varies. On highly predictable text — boilerplate, repetitive formats — patches get large and the latent transformer gets efficient. On information-dense text like dense code or technical writing, patches stay small and the efficiency advantage shrinks.
Training stability at scale is unproven. The BLT paper demonstrates results up to a certain model size. Whether the architecture scales to 70B+ parameters with the same efficiency properties is an open question.
The field hasn’t adopted it yet. LLaMA 4, GPT-5, Gemini 2 all still use tokenizers. BLT is research. The engineering investment to replace tokenization infrastructure at production scale is non-trivial.
The reason BLT is interesting isn’t just the benchmarks.
It’s that tokenization has been treated as an unquestioned given — an implementation detail inherited from the NLP of the early 2010s, carried forward because it worked well enough and changing it was hard.
BLT demonstrates that the “work well enough” part was covering for real weaknesses, and that a principled alternative is achievable. It shows that you can beat a carefully tuned tokenized baseline without a fixed vocabulary, using only 256 byte values and a smarter way of grouping them.
That’s a meaningful result even if BLT itself never ships in a production model.
The broader lesson: the preprocessing stack is not fixed. Tokenization was an engineering convenience that became an invisible constraint. BLT makes the constraint visible — and shows a path around it.
The next generation of models might not start by running your input through a frozen vocabulary lookup. They might start by reading bytes — the way your CPU always has.
In 2020, a group of researchers from JKU Linz published a paper with one of the boldest titles in recent ML history: “Hopfield Networks is All You Need.”
A deliberate nod to the transformer paper. A provocation. And, it turns out, a theorem.
They proved that the update rule of a modern Hopfield network — a theoretical model of associative memory from statistical physics — is mathematically identical to the attention mechanism in transformers. The softmax you use every day is a Hopfield retrieval step. Every attention head is running associative memory.
This is the story of how a 1982 idea about how brains might store memories quietly became the core of the most powerful AI systems in history.
How do you recognize a face you haven’t seen in ten years?
You don’t store a perfect photograph in your head. You store something compressed, something fragmentary. When you see that person again — older, different haircut, different context — your brain fills in the gaps. It retrieves the full memory from a partial cue.
This is associative memory. Not lookup-by-address (like a computer reading memory[0x7FFF]). Lookup-by-content. You give a noisy, incomplete query — and the system finds the closest stored pattern and returns it complete.
Building a mathematical model of this was John Hopfield’s goal in 1982.
Imagine N light switches arranged in a grid. Each switch is either ON (+1) or OFF (-1). Every switch is connected to every other switch by a wire with a weight on it. The whole thing is fully connected, symmetric (the weight from switch A to B equals the weight from B to A), and has no self-connections.
This is the Hopfield network. The state of the system at any moment is the pattern of ON/OFF switches — a vector s = [s1, s2, ..., sN] where each value is either -1 or +1.
Hopfield borrowed an idea from physics: spin glasses. In physics, a spin glass is a magnet where atoms are frozen in random orientations. The system has an energy function that it naturally minimizes over time.
For the Hopfield network, the energy is:
E = -½ × Σ(i≠j) w_ij × s_i × s_j
Think of it this way. If two connected switches i and j have a large positive weight w_ij, and they’re both ON (+1), the product s_i × s_j = +1 contributes a large negative term to E — lowering the energy. The system “likes” when highly-connected switches agree. Energy decreases when the network settles into its preferred configurations.
The key property: every single-switch update either lowers the energy or leaves it unchanged. The network can never get stuck in an endless loop. It always converges.
To store p patterns, you set the weights using the Hebbian rule:
w_ij = (1/N) × Σ(μ=1 to p) [ ξ_i^μ × ξ_j^μ ]
“Neurons that fire together, wire together.” For each stored pattern μ, if neuron i and neuron j are both ON (or both OFF), you strengthen the connection between them. Do this for all patterns, and each pattern carves out a local minimum in the energy landscape.
Think of the energy landscape as a mountainous terrain. Each valley is a stored memory — a stable fixed point the system is attracted to. Corrupted or partial inputs are hikers dropped somewhere on a hillside. The update rule is gravity: the hiker slides downhill and settles in the nearest valley, retrieving the closest stored memory.
Give the network a photo of a face with half the pixels blacked out. It slides down to the nearest energy minimum — the fully-complete stored face. The gap gets filled.
Start from a query (a noisy or partial pattern). Apply the update rule repeatedly:
s_i ← sign( Σ_j w_ij × s_j )
One neuron updates at a time. Each flip lowers E. Eventually you converge to the nearest stored pattern. The network has retrieved a memory from a fragment.
Here’s where classical Hopfield networks hit a wall.
For a network of N neurons trained with Hebbian weights, reliable retrieval works only up to approximately:
C ≈ 0.138 × N
Roughly 14 patterns per 100 neurons. For 1000 neurons, 138 memories max. Beyond this threshold, retrieval collapses catastrophically.
The intuition: when you store a new pattern by strengthening certain connections, you also slightly perturb all existing stored patterns. Each new memory leaves a ghost in every other basin. Up to 0.138N patterns, these ghosts cancel out via averaging. Above it, they accumulate and corrupt retrieval.
A library where every new book smears a tiny bit of ink across every existing book. Up to 138 books per 1000 pages you can still read them. Add the 139th and the smearing cascades into noise.
The second failure mode is subtler and stranger: spurious attractors — energy minima that aren’t any stored pattern.
The most common type is a mixture state. If you store patterns A, B, and C, the combination sign(A + B - C) is often also a stable minimum. It’s not a real memory — it’s a hallucination, a weighted blend of three real ones.
Another guaranteed spurious state: the bitwise inverse of every stored pattern is always also a stable fixed point. The network converges — but to somewhere that was never stored. The hiker finds a phantom valley that shouldn’t exist.
For 30 years, Hopfield networks were mostly taught as historical curiosities. Then three papers rewrote the story:
Krotov & Hopfield (2016): Replace the quadratic interaction in the energy with a higher-order polynomial. Capacity scales as N^(n-1).
Demircigil et al. (2017): Use an exponential interaction e^z. Capacity becomes 2^(N/2) — exponential in N.
Ramsauer et al. (2020): Extend to continuous states (not binary ±1), derive the update rule cleanly, and discover it is exactly transformer attention.
The continuous modern Hopfield network has two competing terms in its energy:
E = -lse(β, X^T × ξ) + ½ × ||ξ||²
The first term — lse, the log-sum-exp function — pulls the query ξ toward stored patterns X. The second term is a regularizer that prevents ξ from running off to infinity to maximize the first.
Log-sum-exp is a smooth version of the max function:
lse(β, z) = (1/β) × log( Σ_i exp(β × z_i) )
At large β, it becomes a hard argmax — pick the single closest pattern. At small β, it averages everything. The β parameter is the sharpness knob.
One step of gradient descent on this energy gives:
ξ_new = X × softmax(β × X^T × ξ)
Unpack this:
X^T × ξ computes the dot-product similarity between the query and every stored patternsoftmax(β × ...) turns these into weights that sum to 1, concentrating weight on the most similar patternsX × [weights] returns a weighted combination of stored patterns — a retrieved memoryThis is not a local flip of one neuron. It’s a global, continuous update that simultaneously considers all stored patterns and returns a smooth interpolation weighted by similarity.
The modern Hopfield network can store approximately 2^(N/2) patterns with exponentially small retrieval error.
For N = 100 dimensions: classical capacity ≈ 14 patterns. Modern capacity ≈ 2^50 ≈ one quadrillion patterns.
The reason: with exponential interaction, each stored pattern carves out a deep, narrow basin. Even when you pack in 2^(N/2) patterns, the basins stay separated. Spurious states essentially vanish because the network almost always converges in a single update step directly to the closest true memory.
Now for the twist.
Generalize the update rule to handle multiple simultaneous queries — a batch of M query vectors processed in parallel:
Ξ_new = X × softmax(β × X^T × Ξ)
Now add learnable linear projections — a standard trick for adding expressive power:
| Hopfield concept | Transformer equivalent |
|---|---|
| Stored patterns X | Keys: K = X × W_K |
| Query states Ξ | Queries: Q = Ξ × W_Q |
| Patterns in output space | Values: V = X × W_V |
| Inverse temperature β | Scaling factor: 1 / sqrt(d_k) |
Substitute all of that in and you get:
Z = softmax( Q × K^T / sqrt(d_k) ) × V
This is the exact transformer self-attention formula. Not similar. Not analogous. Identical.
Every time a transformer runs attention, it is performing one step of Modern Hopfield retrieval. The key matrix is a store of patterns. The query is a partial, noisy signal. The softmax computes similarity scores. The value matrix is the content being retrieved.
The scaling factor 1/sqrt(d_k) is the Hopfield inverse temperature β. This has a direct interpretation for what different transformer layers are doing:
Low β → diffuse softmax → global averaging. The softmax spreads weight across many keys. The retrieved output is a blend of many stored patterns. This is what lower transformer layers do: aggregate broad contextual information.
High β → sharp softmax → single-pattern retrieval. The softmax concentrates almost all weight on the single most-similar key. The output is essentially a copy of that one stored pattern. This is what upper transformer layers do: retrieve specific entities, facts, syntactic structures.
The temperature isn’t fixed — learned attention heads discover the right β for their role in the hierarchy.
Modern transformers operate far below the exponential storage capacity of their attention heads. The number of tokens in context (say, 4096) is astronomically smaller than 2^(d_k/2) for typical d_k = 64 — capacity around 4 billion patterns. The network is running at a fraction of a percent of its storage limit. Every basin is clean. Retrieval converges in one step. Spurious states don’t appear.
The classical library where books smear each other is replaced by a perfect digital index. Each book has its own exact address. One lookup, no cross-contamination.
The Hopfield lens gives new language for mechanistic interpretability.
Every attention head in every layer is storing patterns in its key matrix and retrieving content via its value matrix. The “memories” aren’t fixed — they’re computed fresh from the input on each forward pass. But they behave like memories: the softmax retrieval selects which contextual information to bring forward.
Ramsauer et al. classify transformer behavior into three retrieval regimes:
Global averaging (lower layers): The output is close to the mean of all value vectors. The head is pooling global context. No specific memory retrieved.
Metastable retrieval (middle layers): The output averages a cluster of similar tokens. Semantic grouping — retrieving “all mentions of the subject” rather than one specific occurrence.
Sharp retrieval (upper layers): The output is essentially a single value vector. The head has found an exact match. This is where induction heads, factual recall, and entity tracking happen.
hopfield-layers LibraryRamsauer’s team shipped three PyTorch modules that wrap attention as explicit associative memory:
from hflayers import Hopfield, HopfieldLayer, HopfieldPooling
# Full associative memory — both queries and stored patterns learnable
layer = Hopfield(input_size=256, hidden_size=64, num_heads=8)
# Fixed stored patterns — like a learned prototype bank / lookup table
lookup = HopfieldLayer(input_size=256, quantity=512)
# Single query vector aggregating over variable-length sets
# Replaces CLS token pooling or mean pooling
pooler = HopfieldPooling(input_size=256)
They used HopfieldPooling to classify immune repertoires — sets of hundreds of thousands of antibody sequences — and hit state-of-the-art. The network retrieved diagnostic antibody patterns from a massive noisy pool in a single forward pass.
When Vaswani et al. wrote “Attention is All You Need” in 2017, they described attention as a lookup mechanism. Queries, keys, values. Soft selection. Practical and effective.
What Ramsauer et al. revealed in 2020 is that this “practical mechanism” has a 40-year theoretical foundation in statistical physics and neuroscience. It is not just a useful trick — it is the provably optimal way to do associative memory retrieval with continuous states and exponential capacity.
Every time GPT-4 recalls a fact, every time a code model identifies a variable name, every time a vision transformer detects an object — they are all running a Hopfield retrieval step. Storing patterns in keys, retrieving content from values, computing similarity with softmax.
Hopfield’s 1982 paper was about how brains might work. It turned out to describe, precisely, how the best artificial minds we’ve built actually work.
The 1982 idea wasn’t wrong. It was just 38 years early.
Gemini 1.5 has a 1M token context window. Claude 3.5 handles 200K. GPT-4 Turbo runs 128K. These numbers sound like marketing — until you realize what they actually require in memory.
Every token the model processes stores two vectors in GPU memory: a key and a value, for every attention head, in every layer. These stay resident in VRAM for the entire generation session. With a 70B model running at 128K context, the KV cache alone consumes over 100GB of VRAM — more than most GPU clusters have available.
Long context isn’t free. It’s a memory engineering problem. The techniques that solve it — GQA, quantization, eviction policies, sliding windows — are what separate a research demo from something you can actually deploy.
This post builds from first principles: what the KV cache is and why it matters, then every major compression technique with the intuition before the code.
To understand the cache, you need to understand the cost without it.
When a transformer generates text, it works one token at a time. To generate token number 500, it needs to run attention over all 499 previous tokens — computing how much each prior token should influence the current one. That requires the key and value vectors for all 499 previous tokens.
Here’s the problem: those key and value vectors were already computed when those tokens were first processed. Without a cache, you’d recompute them from scratch every single step. To generate a 1000-token response, you’d compute the keys and values for token 1 a thousand times, for token 2 a total of 999 times, and so on — a staggering amount of redundant work.
The KV cache solves this by storing the key and value vectors as they’re computed, and reusing them on every subsequent step. It’s the difference between O(n²) recomputation and O(n) incremental updates. Without it, even a 4K-token generation would be unbearably slow.
Think of it like a notepad. Every time you read a new sentence in a book, instead of re-reading every previous sentence to remember what happened, you jot down the key facts. The notepad grows with every page — but you never have to go back to re-read.
Here’s a minimal implementation to make it concrete:
import torch
import torch.nn as nn
import torch.nn.functional as F
import math
class AttentionWithKVCache(nn.Module):
def __init__(self, d_model: int, num_heads: int):
super().__init__()
self.num_heads = num_heads
self.d_k = d_model // num_heads
self.W_q = nn.Linear(d_model, d_model, bias=False)
self.W_k = nn.Linear(d_model, d_model, bias=False)
self.W_v = nn.Linear(d_model, d_model, bias=False)
self.W_o = nn.Linear(d_model, d_model, bias=False)
def forward(self, x, kv_cache: dict | None = None):
B, T, C = x.shape
Q = self.W_q(x).view(B, T, self.num_heads, self.d_k).transpose(1, 2)
K = self.W_k(x).view(B, T, self.num_heads, self.d_k).transpose(1, 2)
V = self.W_v(x).view(B, T, self.num_heads, self.d_k).transpose(1, 2)
# If cache exists, prepend previous K/V to current ones
if kv_cache is not None:
if "K" in kv_cache:
K = torch.cat([kv_cache["K"], K], dim=2)
V = torch.cat([kv_cache["V"], V], dim=2)
kv_cache["K"] = K
kv_cache["V"] = V
scale = self.d_k ** -0.5
scores = torch.matmul(Q, K.transpose(-2, -1)) * scale
weights = F.softmax(scores, dim=-1)
out = torch.matmul(weights, V)
out = out.transpose(1, 2).contiguous().view(B, -1, self.num_heads * self.d_k)
return self.W_o(out), kv_cache
The key line is K = torch.cat([kv_cache["K"], K], dim=2) — we’re prepending everything we computed before. The cache just grows, one token at a time, for the entire session.
Here’s where things get alarming. Each cached token stores:
2 vectors (K and V) × num_layers × num_heads × head_dim × bytes_per_float
For Llama 3 70B in fp16 (2 bytes per float):
2 × 80 layers × 64 heads × 128 head_dim × 2 bytes = ~2.5 MB per token
That’s 2.5 megabytes for a single token. Now scale it:
The model weights themselves are only ~140GB. At 128K context, the KV cache is bigger than the model. This is why long context is hard — it’s not an architecture problem, it’s a memory engineering problem.
Every technique in this post is an attack on that 2.5 MB/token number from a different angle.
The most aggressive fix is also the most obvious: instead of storing separate key and value vectors for every attention head, store just one shared pair that all heads use.
In standard multi-head attention (MHA), if you have 64 heads, you have 64 independent K/V projections. Each head learns to attend differently — one head might focus on syntax, another on entity references, another on recency. That diversity is valuable, but it’s expensive: 64 heads means 64× the cache storage.
MQA says: all heads share one K/V. They still each have their own queries (so they can ask different questions), but they all look up the same keys and values. Like a classroom where every student asks different questions, but they’re all reading from the same textbook.
class MultiQueryAttention(nn.Module):
def __init__(self, d_model: int, num_heads: int):
super().__init__()
self.num_heads = num_heads
self.d_k = d_model // num_heads
self.W_q = nn.Linear(d_model, d_model, bias=False) # full — one per head
self.W_k = nn.Linear(d_model, self.d_k, bias=False) # single shared key
self.W_v = nn.Linear(d_model, self.d_k, bias=False) # single shared value
self.W_o = nn.Linear(d_model, d_model, bias=False)
def forward(self, x, kv_cache: dict | None = None):
B, T, _ = x.shape
# Full queries: each head asks its own question
Q = self.W_q(x).view(B, T, self.num_heads, self.d_k).transpose(1, 2)
# Single K and V: shape (B, 1, T, d_k)
K = self.W_k(x).unsqueeze(1)
V = self.W_v(x).unsqueeze(1)
if kv_cache is not None:
if "K" in kv_cache:
K = torch.cat([kv_cache["K"], K], dim=2)
V = torch.cat([kv_cache["V"], V], dim=2)
kv_cache["K"] = K
kv_cache["V"] = V
# Broadcast the single K/V across all query heads
K = K.expand(-1, self.num_heads, -1, -1)
V = V.expand(-1, self.num_heads, -1, -1)
scores = torch.matmul(Q, K.transpose(-2, -1)) / math.sqrt(self.d_k)
out = torch.matmul(F.softmax(scores, dim=-1), V)
out = out.transpose(1, 2).contiguous().view(B, T, -1)
return self.W_o(out), kv_cache
Notice W_k and W_v project to d_k (one head’s worth) instead of d_model (all heads’ worth). The .expand() call broadcasts that single K/V to all heads without actually copying memory — efficient but conceptually it’s one shared lookup.
Memory reduction: 64× on the KV cache for a 64-head model. From 320GB to 5GB at 128K context for Llama 70B. Dramatic.
The cost: measurable quality degradation. When all heads share the same keys and values, they lose their ability to specialize in different types of information. A head that wants to focus on syntactic structure and a head that wants to focus on entity coreference are now forced to look at the same representation — they can ask different questions, but they’re looking at the same answers. Some tasks suffer meaningfully.
Used by: PaLM, Falcon, StarCoder. Mostly superseded by GQA.
MQA went too far. Standard MHA is too expensive. GQA finds the middle ground: instead of one K/V group or 64 K/V groups, use somewhere in between — say, 8 groups for a 64-head model.
Each group of 8 query heads shares one K/V pair. You get 8× memory reduction (vs MHA’s baseline) while giving each group of heads some independence. The heads within a group share a lookup table, but different groups have different lookup tables.
Imagine 64 students split into 8 study groups of 8. Each group shares one set of notes. Students within a group see the same notes, but groups can take different notes. It’s not as rich as everyone having private notes, but it’s much better than the entire class sharing one set.
class GroupedQueryAttention(nn.Module):
def __init__(self, d_model: int, num_heads: int, num_kv_heads: int):
super().__init__()
assert num_heads % num_kv_heads == 0
self.num_heads = num_heads
self.num_kv_heads = num_kv_heads
self.num_queries_per_kv = num_heads // num_kv_heads # heads per group
self.d_k = d_model // num_heads
self.W_q = nn.Linear(d_model, num_heads * self.d_k, bias=False)
self.W_k = nn.Linear(d_model, num_kv_heads * self.d_k, bias=False)
self.W_v = nn.Linear(d_model, num_kv_heads * self.d_k, bias=False)
self.W_o = nn.Linear(d_model, d_model, bias=False)
def forward(self, x, kv_cache: dict | None = None):
B, T, _ = x.shape
Q = self.W_q(x).view(B, T, self.num_heads, self.d_k).transpose(1, 2)
K = self.W_k(x).view(B, T, self.num_kv_heads, self.d_k).transpose(1, 2)
V = self.W_v(x).view(B, T, self.num_kv_heads, self.d_k).transpose(1, 2)
if kv_cache is not None:
if "K" in kv_cache:
K = torch.cat([kv_cache["K"], K], dim=2)
V = torch.cat([kv_cache["V"], V], dim=2)
kv_cache["K"] = K
kv_cache["V"] = V
# Expand each KV group to cover its assigned query heads
# (B, 8 groups, seq, d_k) → (B, 64 heads, seq, d_k)
K = K.repeat_interleave(self.num_queries_per_kv, dim=1)
V = V.repeat_interleave(self.num_queries_per_kv, dim=1)
scores = torch.matmul(Q, K.transpose(-2, -1)) / math.sqrt(self.d_k)
out = torch.matmul(F.softmax(scores, dim=-1), V)
out = out.transpose(1, 2).contiguous().view(B, T, -1)
return self.W_o(out), kv_cache
The critical line is repeat_interleave — it takes group 1’s K/V and assigns it to heads 1–8, group 2’s to heads 9–16, and so on. Only 8 K/V pairs get stored in the cache, not 64.
Real-world impact (Llama 3 70B, 128K context):
| Config | KV Heads | Cache Size |
|---|---|---|
| Standard MHA | 64 | 320 GB |
| GQA (8 KV heads) | 8 | 40 GB |
| MQA | 1 | 5 GB |
40GB is actually manageable on a pair of A100s. Quality versus MHA: barely distinguishable on standard benchmarks. This is why GQA is now the default in every modern open-source model — Llama 2/3, Mistral, Gemma, Falcon 180B, Qwen, Phi. If you’re building a transformer today, GQA is the starting point, not MHA.
GQA reduces the number of vectors stored. Quantization reduces the size of each number in those vectors.
By default, floating point numbers in a transformer are stored in fp16 — 16 bits, or 2 bytes each. The insight behind quantization is that you don’t need full 16-bit precision for every cached value. Most of the information is preserved with 8 bits (int8) or even 4 bits (int4). The numbers get slightly wrong, but not wrong enough to matter for generation quality in most cases.
The process works like this: find the largest absolute value in a vector (the scale), divide everything by that scale to fit it into the smaller integer range, store the integer and the scale separately, then multiply back (dequantize) when you need the actual values. It’s lossy compression — like JPEG for your attention vectors.
class QuantizedKVCache:
def __init__(self, bits: int = 8):
self.bits = bits
self.cache_k: list[torch.Tensor] = []
self.cache_v: list[torch.Tensor] = []
self.scales_k: list[torch.Tensor] = []
self.scales_v: list[torch.Tensor] = []
def _quantize(self, x: torch.Tensor):
# Find the max absolute value per token vector — this becomes our scale
max_val = x.abs().amax(dim=-1, keepdim=True).clamp(min=1e-8)
scale = max_val / (2 ** (self.bits - 1) - 1)
if self.bits == 8:
# Compress float range into -128..127
x_q = (x / scale).round().clamp(-128, 127).to(torch.int8)
else:
# 4-bit: compress into -8..7
x_q = (x / scale).round().clamp(-8, 7).to(torch.int8)
return x_q, scale # store both the compressed value and how to undo it
def _dequantize(self, x_q: torch.Tensor, scale: torch.Tensor):
return x_q.float() * scale # undo compression
def append(self, K: torch.Tensor, V: torch.Tensor):
K_q, scale_k = self._quantize(K)
V_q, scale_v = self._quantize(V)
self.cache_k.append(K_q)
self.cache_v.append(V_q)
self.scales_k.append(scale_k)
self.scales_v.append(scale_v)
def get(self):
# Dequantize everything before attention computation
K = torch.cat([self._dequantize(k, s) for k, s in zip(self.cache_k, self.scales_k)], dim=2)
V = torch.cat([self._dequantize(v, s) for v, s in zip(self.cache_v, self.scales_v)], dim=2)
return K, V
Every token gets quantized as it enters the cache and dequantized just before the attention computation. The GPU stores the compressed integers; the scale factor is tiny (one number per token vector). Net effect: half the memory for int8, quarter for int4.
Memory savings vs fp16:
| Precision | Bytes/value | Reduction |
|---|---|---|
| fp16 (baseline) | 2 | 1× |
| int8 | 1 | 2× |
| int4 | 0.5 | 4× |
| int2 | 0.25 | 8× |
The 2024 KVQuant paper pushed this further with non-uniform quantization — instead of a uniform grid of integer values, it calibrates the grid shape per-channel to match the actual distribution of key/value vectors. Result: fp16-level perplexity at 2-bit precision. That’s a 8× memory reduction with essentially zero quality loss.
Used by: vLLM (int8 KV), llama.cpp, TGI, every major serving framework.
The previous techniques compress how we store tokens. This one asks a different question: do we need to store all tokens at all?
The answer is no — and the reason is one of the most striking empirical findings in LLM research. When you visualize attention weights across a long generation, you find that attention is wildly non-uniform. A tiny fraction of tokens — around 5% — receive roughly 80% of all attention mass. The rest barely get noticed.
These frequently attended tokens are called heavy hitters. They tend to be things like: key facts introduced early in the prompt, entities that the model keeps referencing, structural tokens like paragraph breaks or list markers. The model keeps coming back to them.
The remaining 95% of tokens get attended to occasionally or never. Evicting them would barely change the model’s behavior — but it would dramatically shrink the cache.
H2O (Heavy-Hitter Oracle) implements this idea: track cumulative attention scores for every cached token. When the cache exceeds a budget, evict the tokens with the lowest cumulative scores. Always keep the most recent token regardless (since it’s needed for immediate context).
class H2OKVCache:
def __init__(self, max_cache_size: int, num_heads: int, d_k: int):
self.max_cache_size = max_cache_size
self.num_heads = num_heads
self.d_k = d_k
self.K = None
self.V = None
# Running tally: how much total attention has each token received?
self.attention_scores = None
def update(self, K_new, V_new, attn_weights):
"""
Called each generation step.
attn_weights: how much the new token attended to each cached token.
We use this to update each cached token's importance score.
"""
if self.K is None:
self.K = K_new
self.V = V_new
self.attention_scores = torch.zeros(K_new.shape[0], self.num_heads, 1, device=K_new.device)
return
# Add this step's attention to the running total for each token
self.attention_scores += attn_weights.sum(dim=2, keepdim=True).transpose(2, 3)
# Append new token to cache
self.K = torch.cat([self.K, K_new], dim=2)
self.V = torch.cat([self.V, V_new], dim=2)
new_score = torch.zeros(K_new.shape[0], self.num_heads, 1, device=K_new.device)
self.attention_scores = torch.cat([self.attention_scores, new_score], dim=2)
# Over budget? Evict the least important tokens
if self.K.shape[2] > self.max_cache_size:
self._evict()
def _evict(self):
seq_len = self.K.shape[2]
keep = self.max_cache_size
recent_idx = seq_len - 1
# Average importance across all heads
importance = self.attention_scores.mean(dim=1).squeeze(1) # (batch, seq)
# Protect the most recent token — always needed
scores_no_recent = importance.clone()
scores_no_recent[:, recent_idx] = float('-inf')
# Take the (keep-1) highest-importance historical tokens, plus the recent one
_, top_indices = scores_no_recent.topk(keep - 1, dim=-1)
keep_indices = torch.cat([
top_indices,
torch.tensor([[recent_idx]], device=top_indices.device).expand(top_indices.shape[0], -1)
], dim=-1)
keep_indices, _ = keep_indices.sort(dim=-1)
# Discard the evicted tokens from K, V, and score trackers
idx = keep_indices.unsqueeze(1).unsqueeze(-1).expand(-1, self.num_heads, -1, self.d_k)
self.K = torch.gather(self.K, 2, idx)
self.V = torch.gather(self.V, 2, idx)
self.attention_scores = torch.gather(
self.attention_scores, 2,
keep_indices.unsqueeze(1).expand(-1, self.num_heads, -1)
)
The _evict method is the core: sort tokens by cumulative attention received, keep the top-K, drop the rest. The recently added token is always protected from eviction because it hasn’t had time to accumulate attention scores yet.
H2O results: 20× cache compression with less than 1% accuracy degradation on most benchmarks. Set a cache budget of 5% of your sequence length and you barely notice the difference on most tasks.
Why? Because the model was already ignoring 95% of the tokens. Evicting them just makes the neglect explicit.
H2O evicts the least-attended tokens. But researchers at MIT made a more fundamental observation: not all tokens are equal from the moment they’re created.
When you visualize attention patterns across thousands of generation steps, two patterns appear consistently:
Pattern 1 — Locality: most attention concentrates on the last few hundred tokens. The model cares most about what it just said. This is intuitive — you need your immediate context to write coherently.
Pattern 2 — Attention sinks: the very first tokens in any sequence (positions 0, 1, 2, 3) receive surprisingly high attention throughout the entire generation — even when their content is completely irrelevant to the current generation step. A document about quantum physics will still have heavy attention on its first four tokens, even 50,000 tokens later.
This second pattern is counterintuitive and strange. Why would a model keep attending to the first few words of a document it’s now thousands of tokens past?
The answer is a training artifact. During training, the softmax over attention scores must sum to 1. When a token has nothing relevant to attend to, the probability mass has to go somewhere. The model learned to dump it onto the first few tokens as a kind of “soft null” — they absorb the leftover probability. They’re not semantically important, they’re structurally necessary as garbage collectors for the softmax.
StreamingLLM uses both observations to build a fixed-size cache for infinite-length generation: always keep the first 4 tokens (the sinks) plus a rolling window of the most recent N tokens. Everything between the sink tokens and the window gets dropped permanently.
class StreamingKVCache:
def __init__(self, sink_size: int = 4, window_size: int = 1020):
"""
Total cache = sink_size + window_size (constant, regardless of sequence length).
The sink tokens anchor the softmax. The window provides local context.
Everything older than the window is evicted.
"""
self.sink_size = sink_size
self.window_size = window_size
self.sink_K = self.sink_V = None
self.window_K = self.window_V = None
def update(self, K_new: torch.Tensor, V_new: torch.Tensor):
# Phase 1: fill the sink buffer with the first few tokens
if self.sink_K is None or self.sink_K.shape[2] < self.sink_size:
self.sink_K = torch.cat([self.sink_K, K_new], dim=2) if self.sink_K is not None else K_new
self.sink_V = torch.cat([self.sink_V, V_new], dim=2) if self.sink_V is not None else V_new
return self.get()
# Phase 2: everything after the sink goes into the rolling window
if self.window_K is None:
self.window_K = K_new
self.window_V = V_new
else:
self.window_K = torch.cat([self.window_K, K_new], dim=2)
self.window_V = torch.cat([self.window_V, V_new], dim=2)
# Window full — drop the oldest token (slide forward)
if self.window_K.shape[2] > self.window_size:
self.window_K = self.window_K[:, :, 1:, :]
self.window_V = self.window_V[:, :, 1:, :]
return self.get()
def get(self):
if self.window_K is None:
return self.sink_K, self.sink_V
return (
torch.cat([self.sink_K, self.window_K], dim=2),
torch.cat([self.sink_V, self.window_V], dim=2),
)
The window_K[:, :, 1:, :] line is the sliding — every new token pushes the oldest window token off the edge. But the sink tokens never move.
What happens without the sinks? If you just use a sliding window with no sink tokens, the model collapses — perplexity spikes dramatically after the context fills. The softmax has nowhere to send its garbage probability mass and the distribution breaks down. The 4 sink tokens are tiny in cost but critical in function.
StreamingLLM result: Llama 2 ran on 4-million-token documents using a 1024-token cache. 22× memory reduction. Constant memory usage no matter how long the generation runs.
Used by: Mistral (sliding window attention), Gemma, Longformer.
All the techniques above deal with the cache during generation — either evicting tokens as you go or using a fixed window. SnapKV takes a step back and asks: can we compress the cache before generation even starts?
When an LLM processes a long prompt (say, 50,000 tokens of context before generating an answer), it runs what’s called a prefill step — a full forward pass over the entire prompt in parallel. During this prefill, every token attends to every other token, producing a full attention matrix.
The SnapKV insight: that prefill attention matrix tells you which tokens are important before generation begins. Tokens that many other tokens attend to during prefill are likely to stay important during decoding. You can use the prefill attention to pre-select which tokens to keep in the cache, and discard the rest before decoding starts.
def snapkv_compress(K, V, attn_weights, compression_ratio=0.3):
"""
Run this after prefill, before generation starts.
We look at how much attention each token received from others during prefill.
High-attention tokens are "important" — keep them.
Low-attention tokens are likely padding or filler — discard them.
compression_ratio = 0.3 means keep only the top 30% of tokens.
"""
B, H, S, d_k = K.shape
keep_count = max(1, int(S * compression_ratio))
# Column sums of attention matrix: how much did each token receive?
# Shape: (B, H, S) — one score per token per head
importance = attn_weights.sum(dim=2)
# Average importance across all heads to get a single score per token
avg_importance = importance.mean(dim=1) # (B, S)
# Pick the top-k most attended-to tokens
_, top_indices = avg_importance.topk(keep_count, dim=-1)
top_indices, _ = top_indices.sort(dim=-1) # preserve original order
# Build the compressed cache — only these tokens enter generation
idx = top_indices.unsqueeze(1).unsqueeze(-1).expand(B, H, -1, d_k)
K_compressed = torch.gather(K, 2, idx)
V_compressed = torch.gather(V, 2, idx)
return K_compressed, V_compressed
The key idea is attn_weights.sum(dim=2) — summing across the “who is attending” dimension to get “how much is each token being attended to.” High score = many tokens find this one important = keep it. Low score = nobody’s looking at this = safe to discard.
SnapKV results: 3.6× cache reduction, 3.8× speedup on generation, less than 1% accuracy loss on LongBench. Particularly strong for RAG-style tasks where the prompt is mostly retrieved context — most chunks aren’t relevant to the answer, and SnapKV confidently discards them based on prefill attention.
The difference from H2O: SnapKV makes one compression decision upfront (at prefill time) and never changes it. H2O continuously evicts during generation. SnapKV is simpler and faster; H2O adapts dynamically as the generation unfolds.
These aren’t competing approaches — they’re layers. Production systems combine them:
GQA → reduces number of K/V vectors stored
+ INT8 quant → halves bytes per stored value
+ H2O eviction → further reduces token count dynamically
+ Sliding window → caps total cache at constant size
Here’s what that stacking does for Llama 3 70B at 128K context:
| What’s applied | Cache Size |
|---|---|
| Baseline MHA, fp16 | 320 GB |
| + GQA (8 KV heads) | 40 GB |
| + INT8 quantization | 20 GB |
| + H2O (20% budget) | 4 GB |
From 320 GB to 4 GB. Same model. Same outputs (approximately). Now fits on a single A100 with room for the model weights too.
Smaller cache isn’t just about fitting in memory — it directly affects how fast the model generates tokens.
Every time the model generates a new token, it reads the entire KV cache from GPU high-bandwidth memory (HBM). The speed of that read is bounded by HBM bandwidth — for an A100, that’s about 2 TB/s. At 128K context with GQA 70B:
Apply H2O + quantization to bring cache to 4 GB:
A 10× throughput gain from the same GPU, same model, same weights. This is why inference frameworks like vLLM, TensorRT-LLM, and SGLang have invested so heavily in KV cache engineering — it’s not just about memory, it’s about the money you spend on compute per generated token.
KV cache is the dominant memory cost at long context — at 128K tokens, the cache is bigger than the model weights. This is the problem every technique here is solving.
GQA is the new baseline — 8× memory reduction, negligible quality loss. Every serious model uses it now. MHA is legacy.
Quantization stacks cleanly on top of GQA — INT8 gives you 2× more, INT4 gives 4× more, 2-bit is viable with per-channel calibration.
Most tokens don’t matter — 80% of attention concentrates on 5% of tokens. Eviction strategies (H2O, SnapKV) exploit this to compress aggressively without hurting quality.
Attention sinks are a real phenomenon — the first few tokens in any sequence must always be kept. Removing them breaks generation even when their content is irrelevant. This is a training artifact, not a design choice.
Cache size = latency — HBM bandwidth is the bottleneck during decoding. Smaller cache means faster reads means more tokens per second. Memory compression and speed optimization are the same problem.
Efficient Memory Management for Large Language Model Serving with PagedAttention — Kwon et al., UC Berkeley (2023) The vLLM paper. Introduces virtual memory paging for KV cache — eliminates fragmentation, enables dynamic batching across requests. The production standard for KV cache management.
Fast Transformer Decoding: One Write-Head is All You Need (MQA) — Shazeer, Google (2019) The original MQA paper. Shows that sharing KV heads barely hurts quality while massively reducing memory and improving decode speed. The idea GQA refined.
GQA: Training Generalized Multi-Query Transformer Models from Multi-Head Checkpoints — Ainslie et al., Google (2023) GQA paper. Introduces the grouped approach and shows how to convert existing MHA checkpoints via uptraining. The technique behind Llama 2, Mistral, and every modern open-source model.
KVQuant: Towards 10 Million Context Length LLM Inference with KV Cache Quantization — Hooper et al., UC Berkeley (2024) Non-uniform quantization calibrated per-channel. Achieves fp16 perplexity at 2-bit precision. The paper that proved INT2 KV cache is viable.
KIVI: A Tuning-Free Asymmetric 2-bit Quantization for KV Cache — Liu et al. (2024) Keys and values have different numerical distributions — KIVI quantizes them differently. Clean result: 2-bit KV cache with almost no accuracy loss, no fine-tuning required.
H2O: Heavy-Hitter Oracle for Efficient Generative Inference of Large Language Models — Zhang et al., Rice University (2023) The paper that identified the heavy-hitter phenomenon. 20× cache compression with less than 1% degradation. Empirically shows that 80% of attention concentrates on 5% of tokens.
SnapKV: LLM Knows What You are Looking for Before Generation — Li et al. (2024) Prefill-time compression. Uses the attention matrix from prompt processing to predict which tokens will matter during decoding. 3.6× compression, 3.8× speedup.
Efficient Streaming Language Models with Attention Sinks (StreamingLLM) — Xiao et al., MIT (2023) Discovers and explains attention sinks. Shows that 4 sink tokens + rolling window enables infinite-length generation at constant memory. Clean, surprising, and practically useful.
FlashAttention-2: Faster Attention with Better Parallelism and Work Partitioning — Dao (2023) IO-aware attention that tiles computation to avoid materializing the full O(n²) attention matrix. The algorithm that makes long-context attention practical from a compute perspective — complements KV cache compression from the memory side.
Sarathi-Serve: Efficient LLM Inference by Piggybacking Decodes with Chunked Prefills — Agrawal et al. (2023) Production serving paper on pipelining prefill and decode efficiently. The operational reality of running bounded KV cache systems at scale.
The context window arms race isn’t about model capability — it’s about memory engineering. Every extra token you can fit in context at the same VRAM budget is a capability gain. The teams winning on long context aren’t necessarily building smarter models. They’re building smarter caches.
]]>Every demo of an AI agent shows the same thing: the agent calls a tool, gets data back, generates an answer. What nobody shows is what that tool server actually looks like and how to build one yourself.
The Model Context Protocol (MCP) is the emerging standard for exactly this — a clean interface between AI agents and the tools/APIs they need to interact with. Think of it as the HTTP for agent-tool communication. The agent doesn’t need to know how your weather API works internally; it just needs to know what tools exist and how to call them.
This post walks through building an MCP-style server from scratch with Python and FastAPI — the minimal version that teaches you the pattern, plus the upgrade path to production.
AI Agent → MCP Server → Tools / External APIs
Three actors:
The agent doesn’t hallucinate tool results because it gets them from your server. Your server doesn’t need to understand the agent because it just responds to HTTP requests. Clean separation.
pip install fastapi uvicorn requests
That’s the full dependency list for the starter. FastAPI handles routing and serialization; Uvicorn is the ASGI server.
mcp-server/
├── main.py
└── README.md
Start flat. Add structure when you have more than ~5 tools.
from fastapi import FastAPI
app = FastAPI()
@app.get("/tool/weather")
def weather(city: str):
return {
"tool": "weather",
"city": city,
"temp": "32°C",
"status": "Sunny"
}
@app.get("/tool/users")
def users():
return {
"tool": "users",
"data": [
{"id": 1, "name": "Rahul"},
{"id": 2, "name": "Priya"}
]
}
@app.get("/tools")
def tools():
return {
"tools": [
{"name": "weather", "endpoint": "/tool/weather?city=Delhi"},
{"name": "users", "endpoint": "/tool/users"}
]
}
Run it:
uvicorn main:app --reload
Server up at http://127.0.0.1:8000.
/toolsGET /tools
This is the contract between agent and server. The agent hits this endpoint first, gets back a list of available tools and their call signatures. No hardcoding required — the agent adapts to whatever tools your server advertises.
/tool/weatherGET /tool/weather?city=Hyderabad
Response:
{
"tool": "weather",
"city": "Hyderabad",
"temp": "32°C",
"status": "Sunny"
}
/tool/usersGET /tool/users
Response:
{
"tool": "users",
"data": [
{ "id": 1, "name": "Rahul" },
{ "id": 2, "name": "Priya" }
]
}
The agent loop looks like this:
GET /tools → parse available tool names and endpointsNo magic. No framework required. Any agent that can make HTTP requests can use this server — Claude via tool use, GPT-4 function calling, a custom ReAct loop, anything.
The starter above is intentionally minimal. Here’s where to take it:
Auth layer — add API key validation via FastAPI’s Depends:
from fastapi import Header, HTTPException
async def verify_api_key(x_api_key: str = Header(...)):
if x_api_key != "your-secret-key":
raise HTTPException(status_code=401, detail="Invalid API key")
Real data — swap the hardcoded returns for actual API calls or database queries:
import httpx
@app.get("/tool/weather")
async def weather(city: str):
async with httpx.AsyncClient() as client:
resp = await client.get(
"https://api.openweathermap.org/data/2.5/weather",
params={"q": city, "appid": WEATHER_API_KEY, "units": "metric"}
)
data = resp.json()
return {
"tool": "weather",
"city": city,
"temp": f"{data['main']['temp']}°C",
"status": data["weather"][0]["description"],
}
Memory layer — add a simple in-memory store (or Redis) so agents can persist state across calls:
from collections import defaultdict
memory: dict[str, list] = defaultdict(list)
@app.post("/memory/{session_id}")
def store(session_id: str, payload: dict):
memory[session_id].append(payload)
return {"stored": True}
@app.get("/memory/{session_id}")
def recall(session_id: str):
return {"history": memory[session_id]}
Logging — every tool call should log (timestamp, tool_name, params, response_time). Non-negotiable in production. Agents fail silently without it.
Same pattern in every case: define the tool, expose it as an endpoint, register it in /tools. The agent handles the reasoning; your server handles the execution.
MCP is just HTTP with a discovery contract — /tools tells the agent what exists; individual endpoints do the work. No exotic protocol needed to start.
Tool discovery is the key endpoint — it’s what makes your server agent-agnostic. Any LLM that can read JSON can use your tools without custom integration.
Keep tools single-responsibility — one endpoint, one job. Agents get confused by tools that do too much. /tool/weather does weather. That’s it.
Auth before production — the starter has no auth. Add API key verification before exposing any real data.
Logging is observability — you can’t debug a multi-agent system without knowing what tool was called, with what params, and what came back.
Model Context Protocol Spec — Anthropic’s official MCP specification. The standard this starter is inspired by — covers the full protocol including resources, prompts, and sampling beyond just tools.
FastAPI Docs — Best Python API framework for this use case. Dependency injection, auto-generated OpenAPI docs, and async support out of the box.
Building Effective Agents — Anthropic’s guide on agent patterns. Covers tool use, ReAct loops, and when to use multi-agent vs. single-agent architectures.
LangChain Tools — If you want to integrate your MCP server with an existing agent framework, LangChain’s tool abstraction maps cleanly onto this pattern.
The agent is the brain. Your MCP server is the hands. Getting the interface between them right — clean endpoints, honest tool descriptions, fast responses — is what separates a demo from a system.
]]>What if you could simulate a neuroscience experiment before running it on a single human subject?
That’s the quiet but radical promise buried inside a recent paper: “A foundation model of vision, audition, and language for in-silico neuroscience.”
The model is called TRIBE v2. It takes video, audio, and text simultaneously, predicts whole-brain fMRI responses across 720 subjects and 1,000+ hours of data — and then, in its most striking result, reproduces classic neuroscience findings without running a single new human experiment.
That last part is what makes this paper worth a deep read.
Most brain models are narrow by design.
One model handles visual cortex. Another handles auditory cortex. A third tries to align language models with text-driven activity. Each is tuned to a specific task, dataset, and usually a handful of subjects.
That fragmentation has produced real science — but it also creates a fundamental mismatch. The brain doesn’t process the world in academic silos. It watches, listens, reads, and integrates — simultaneously, continuously, in real time.
Meanwhile, every new experiment still requires:
The field moves slowly because iteration is expensive. TRIBE v2 asks: what if a strong enough computational model could compress that loop?
TRIBE v2 is built on a clean architectural principle: pretrained modality encoders already learned the world. Teach them to map into brain space.
Instead of handcrafting neuroscience-specific features, the authors take embeddings from state-of-the-art pretrained models for video, audio, and language. Those embeddings are time-synchronized, concatenated into a shared multimodal representation, and passed through a trainable transformer that predicts fMRI responses across the whole brain.
The pipeline:
Stimulus (video + audio + text)
↓
Pretrained modality encoders (separate per stream)
↓
Time-aligned multimodal embeddings
↓
Transformer brain encoder (the trainable part)
↓
Predicted whole-brain fMRI activity
This is a major shift from voxel-wise encoding pipelines — the standard approach — where researchers train separate linear models per subject, per task, and per brain region. TRIBE v2 learns one general model that handles all of them.
Scale is not a footnote here. It’s the foundation.
| Dataset type | Details |
|---|---|
| Total fMRI data | 1,000+ hours |
| Subjects | 720 individuals |
| Dataset types | Deep (many sessions/subject) + Wide (many subjects, fewer sessions) |
| Stimulus types | Movies, podcasts, multimodal narratives |
Aggregating this much fragmented neuroscience data into one modeling setup is itself a contribution. No individual lab has collected anything close to this in one place.
When TRIBE v2 predicts brain responses to audio-heavy stimuli, activations peak in temporal auditory regions. For video-heavy stimuli, they peak in visual cortex. For multimodal inputs, predictions recruit broad cortical networks.
That sounds like it should just work automatically — but it doesn’t in naive multimodal fusion. A poorly designed joint model blurs modality-specific responses into meaningless soup. The fact that TRIBE v2 preserves clean structure while still benefiting from cross-modal context is a meaningful result.
The authors compare against Deep FIR — an optimized linear encoding pipeline using the same pretrained embeddings. This is the honest comparison. It isolates whether the transformer architecture is doing anything beyond better input features.
It is. TRIBE v2 outperforms the linear baseline significantly across datasets, meaning the nonlinear cross-modal interactions are genuinely adding value — not just free-riding on stronger representations.
This is the most practically significant result.
TRIBE v2 can predict brain responses for participants it has never seen, in a zero-shot setting. For some datasets, its group-level predictions beat what you’d get from a typical individual subject measured against the cohort average.
Think about what that enables:
The model doesn’t replace human data. But it could front-load the iteration to before you collect any.
With even a small amount of subject-specific data, fine-tuning improves performance substantially. The workflow mirrors what foundation models did for NLP and vision:
No training from scratch. No waiting for a full dataset.
Here’s where the paper earns its headline claim.
The authors test whether TRIBE v2 can reproduce findings from classic neuroscience experiments using only synthetic model-generated predictions — no new human subjects, no scanner.
They simulate visual functional localizer experiments using controlled categories: faces, places, bodies, written characters.
TRIBE v2 recovers the expected brain regions:
| Category | Brain region recovered |
|---|---|
| Faces | FFA — Fusiform Face Area |
| Places | PPA — Parahippocampal Place Area |
| Bodies | EBA — Extrastriate Body Area |
| Words | VWFA — Visual Word Form Area |
These are landmarks of cognitive neuroscience, established over decades of carefully controlled human experiments. A model reproducing them from simulated predictions is not just passing a benchmark — it’s demonstrating structural alignment with the field’s validated knowledge.
The same approach extends to language. The model is tested on conditions designed to isolate language processing, and the predicted contrast maps align with known language network responses — even though language unfolds over time, interacts with audio, and depends on hierarchical syntactic structure.
If in-silico experiments become reliable enough, neuroscience research gets a new design loop:
Hypothesis
↓
Simulate expected brain response with a foundation model
↓
Refine stimulus or experimental protocol
↓
Run the expensive human experiment (now with higher confidence)
Every other field has a version of this. Aerospace has wind tunnels. Drug discovery has molecular simulations. Physics has lattice QCD. Neuroscience has mostly had the scanner and hope.
TRIBE v2 is a first serious hint that computational simulation could become a standard upstream step in brain research.
Using independent component analysis, the authors probe what the model’s latent space actually captures.
The components align recognizably with:
They also find something interesting: text-derived features dominate not just classical language areas but parts of prefrontal cortex too — suggesting that linguistic abstraction may provide a useful organizational scaffold for broader cognitive structure.
That’s a scientific finding, not just a benchmark score.
The authors don’t oversell this. A few things to keep in mind:
TRIBE v2 models the brain as a passive observer. It predicts responses to stimuli. It says nothing about active decision-making, motor planning, or internally generated thought.
fMRI trades temporal resolution for spatial coverage. The model is trained on a signal that averages over 1–2 seconds of neural activity. Real brain computation is happening at milliseconds. That’s a massive compression.
Strong prediction ≠ mechanism. TRIBE v2 may be a very powerful statistical emulator of brain responses without capturing the true computational principles underneath. The map is not the territory.
Data diversity is still a gap. 720 subjects is large for neuroscience — but the population is still almost certainly skewed toward Western, educated, young, healthy adults. A “foundation model of the brain” is only as universal as the brains it was trained on.
TRIBE v2 does not try to reinvent the foundation-model stack for neuroscience. It borrows what already worked:
That’s pragmatic. The best audio, video, and language representations are already being learned at internet scale. Neuroscience doesn’t need to redo that. It just needs a bridge between those representations and brain space.
From an engineering standpoint, this also explains why the model scales well. Once the modality encoders are strong, the remaining problem is multimodal integration, temporal aggregation, and subject-aware decoding — and transformers are already excellent at all three.
The strongest papers don’t just improve a metric. They change the default question.
Before work like this, the standard neuroscience framing was:
“Can we build a model for this specific experiment?”
After TRIBE v2, the framing becomes:
“Can we build a general computational instrument that makes experiments better before we run them?”
That is a more ambitious goal, and a more interesting one.
Language models became interfaces to knowledge. Diffusion models became interfaces to visual generation. Foundation models like TRIBE v2 may become interfaces to hypothesis generation in neuroscience — a tool you run before you commit to months of experimental design.
TRIBE v2 is not claiming to have solved the brain. It’s more honest than that.
What it’s claiming — and largely demonstrating — is that a general-purpose model trained on enough multimodal brain data can do something genuinely new: simulate neuroscience before you run it.
If that workflow matures, it could do for cognitive neuroscience what molecular simulation did for drug discovery and what CFD did for aerodynamics: compress the iteration cycle and expand the space of questions researchers can actually afford to ask.
That’s worth paying attention to.
When Mistral released Mixtral 8x7B in December 2023, people were confused. The model card said 46.7B total parameters but only 12.9B active per token. How does a 46B model run at the speed of a 13B one?
When DeepSeek-V3 launched in late 2024 with 671B total parameters but only 37B active, the confusion deepened. Numbers that large shouldn’t fit on any reasonable hardware — but they did.
The answer is Mixture of Experts (MoE): a decades-old idea from the early 1990s that became the dominant architecture for frontier LLMs once engineers figured out how to make it work at scale.
The core insight is almost offensively simple: you don’t need to run every part of the network on every input. Different tokens need different computations. A model smart enough to route each token to the relevant specialists — and ignore the rest — gets the knowledge of a huge model at the cost of a small one.
This post goes deep on how that works: the router, the expert layers, the training tricks, and the engineering tradeoffs that make production MoE systems possible.
In a standard transformer, every token passes through every parameter on every forward pass. A 70B dense model activates all 70B parameters for each token — a word in a poem, a number in an equation, a variable name in code. The model applies the full machinery regardless of whether it’s relevant.
This is wasteful. A token in a Python function doesn’t need the same computations as a token in a French poem. The model’s knowledge of French grammar is entirely irrelevant when parsing def forward(self, x):.
Dense scaling has another problem: compute scales with parameters. Double the parameters, roughly double the FLOPs per token. At some point, training cost becomes the bottleneck — not capability.
MoE breaks this link between total parameters and per-token compute.
In a standard transformer block, the feed-forward network (FFN) is the largest sub-component — typically 4× the model dimension, applied identically to every token.
MoE replaces this single FFN with N independent FFNs — the experts — plus a lightweight router that selects which experts handle each token:
Standard Transformer Block:
x → Self-Attention → Add & Norm → FFN → Add & Norm → output
MoE Transformer Block:
x → Self-Attention → Add & Norm → Router → Top-K Experts → Add & Norm → output
The self-attention layers stay dense — every token attends to every other token. Only the FFN is replaced. This is the key design choice: attention handles context and relationships; experts handle token-level transformation.
The router is a small linear layer that maps each token’s hidden state to a probability distribution over experts:
import torch
import torch.nn as nn
import torch.nn.functional as F
class MoERouter(nn.Module):
def __init__(self, hidden_dim: int, num_experts: int, top_k: int = 2):
super().__init__()
self.num_experts = num_experts
self.top_k = top_k
# A single linear layer: hidden_dim → num_experts logits
self.gate = nn.Linear(hidden_dim, num_experts, bias=False)
def forward(self, x):
"""
x: (batch_size * seq_len, hidden_dim)
Returns:
top_k_indices: (batch_size * seq_len, top_k) — which experts to use
top_k_weights: (batch_size * seq_len, top_k) — how much to weight each
"""
# Compute logits over all experts
logits = self.gate(x) # (tokens, num_experts)
# Softmax to get routing probabilities
probs = F.softmax(logits, dim=-1) # (tokens, num_experts)
# Select top-K experts per token
top_k_weights, top_k_indices = torch.topk(probs, self.top_k, dim=-1)
# Renormalize so selected weights sum to 1
top_k_weights = top_k_weights / top_k_weights.sum(dim=-1, keepdim=True)
return top_k_indices, top_k_weights
For each token, the router outputs two things: which K experts to use, and how much weight to give each one. The final output is a weighted sum of those K experts’ outputs.
Typical values: num_experts = 8, top_k = 2. So each token uses exactly 2 of 8 experts — activating 25% of the expert parameters.
Each expert is a standard feed-forward network — nothing exotic:
class Expert(nn.Module):
def __init__(self, hidden_dim: int, ffn_dim: int):
super().__init__()
self.net = nn.Sequential(
nn.Linear(hidden_dim, ffn_dim),
nn.SiLU(), # SwiGLU variant used in Mixtral
nn.Linear(ffn_dim, hidden_dim),
)
def forward(self, x):
return self.net(x)
The size of each expert’s FFN is typically the same as the FFN in a comparable dense model. So Mixtral 8x7B has 8 experts, each the size of a 7B model’s FFN — but only 2 are active per token. The “8x7B” name reflects this: 8 experts, each ~7B FFN scale.
class MoELayer(nn.Module):
def __init__(self, hidden_dim: int, ffn_dim: int, num_experts: int, top_k: int = 2):
super().__init__()
self.num_experts = num_experts
self.top_k = top_k
self.router = MoERouter(hidden_dim, num_experts, top_k)
self.experts = nn.ModuleList([
Expert(hidden_dim, ffn_dim) for _ in range(num_experts)
])
def forward(self, x):
"""
x: (batch_size, seq_len, hidden_dim)
"""
B, S, H = x.shape
# Flatten to (batch*seq, hidden) — router operates per token
x_flat = x.view(B * S, H)
# Get routing decisions
expert_indices, expert_weights = self.router(x_flat)
# expert_indices: (B*S, top_k)
# expert_weights: (B*S, top_k)
# Collect expert outputs
output = torch.zeros_like(x_flat)
for k in range(self.top_k):
# Which tokens go to this slot's expert?
indices = expert_indices[:, k] # (B*S,) — expert index per token
weights = expert_weights[:, k] # (B*S,) — weight per token
# Process each expert's assigned tokens
for expert_id in range(self.num_experts):
token_mask = (indices == expert_id)
if not token_mask.any():
continue
expert_input = x_flat[token_mask]
expert_output = self.experts[expert_id](expert_input)
# Weighted accumulation
output[token_mask] += weights[token_mask].unsqueeze(-1) * expert_output
return output.view(B, S, H)
This is the naive implementation — readable but slow. Production systems batch all tokens destined for the same expert together (a single large matmul instead of a loop), which is why MoE requires specialized CUDA kernels in practice.
Here’s the failure mode that makes MoE training notoriously tricky: expert collapse.
Without any intervention, the router quickly learns to always route tokens to the same 1–2 experts. Those experts see more gradients, get better faster, and attract even more tokens. The remaining experts starve — they’re never trained because tokens never reach them. You end up with a 141B model that effectively uses 10B parameters.
The standard fix is an auxiliary load balancing loss that penalizes unequal expert utilization:
def load_balancing_loss(router_probs, expert_indices, num_experts):
"""
Encourages uniform distribution of tokens across experts.
router_probs: (num_tokens, num_experts) — softmax output of router
expert_indices: (num_tokens, top_k) — selected expert indices
"""
num_tokens = router_probs.shape[0]
# Fraction of tokens routed to each expert
# (how often each expert is selected)
one_hot = F.one_hot(expert_indices, num_classes=num_experts).float()
tokens_per_expert = one_hot.sum(dim=1).mean(dim=0) # (num_experts,)
f = tokens_per_expert / tokens_per_expert.sum()
# Average router probability assigned to each expert
# (how confident the router is about each expert)
p = router_probs.mean(dim=0) # (num_experts,)
# Loss = num_experts * sum(f_i * p_i)
# Minimized when both f and p are uniform (1/num_experts each)
loss = num_experts * (f * p).sum()
return loss
This loss is added to the main language modeling loss with a small coefficient (typically α = 0.01). It gently pushes the router toward distributing tokens evenly without overwhelming the task objective.
Even with load balancing, real deployments face a harder constraint: expert capacity.
In distributed training, each expert lives on a specific device. If 1000 tokens all want expert #3 and only 100 want expert #7, expert #3’s device is overwhelmed while #7’s sits idle. You can’t dynamically resize — the devices are fixed.
The solution is a capacity factor: set a maximum number of tokens each expert can process per forward pass. Tokens that exceed an expert’s capacity are dropped — they don’t get processed by any expert.
def route_with_capacity(router_probs, top_k, capacity_factor=1.25):
"""
Routes tokens to experts with a hard capacity ceiling.
Excess tokens are dropped (their expert output is zeroed out).
"""
num_tokens, num_experts = router_probs.shape
# Max tokens any single expert will process
capacity = int(capacity_factor * num_tokens * top_k / num_experts)
top_k_weights, top_k_indices = torch.topk(router_probs, top_k, dim=-1)
top_k_weights = top_k_weights / top_k_weights.sum(dim=-1, keepdim=True)
# Track how many tokens each expert has accepted
expert_counts = torch.zeros(num_experts, dtype=torch.long)
dispatch_mask = torch.zeros(num_tokens, top_k, dtype=torch.bool)
for token_idx in range(num_tokens):
for k in range(top_k):
expert_id = top_k_indices[token_idx, k].item()
if expert_counts[expert_id] < capacity:
dispatch_mask[token_idx, k] = True
expert_counts[expert_id] += 1
# else: token is dropped for this expert slot
return top_k_indices, top_k_weights, dispatch_mask
Token dropping introduces a tricky training dynamic: the model must learn to be robust to occasionally missing expert outputs. In practice, capacity factor ~1.25 means ~2–5% of tokens get dropped — small enough that the model learns to tolerate it.
Mixtral 8x7B uses 8 experts per MoE layer with top-2 routing. Every other transformer layer is an MoE layer (the rest are standard dense attention + FFN). The architecture:
| Property | Value |
|---|---|
| Total parameters | 46.7B |
| Active parameters per token | 12.9B |
| Experts per MoE layer | 8 |
| Active experts per token | 2 |
| Hidden dimension | 4096 |
| Expert FFN dimension | 14336 |
| Layers | 32 (alternating dense/MoE) |
The critical insight: attention layers are dense (all 12.9B active parameters), but the expert FFNs represent the bulk of “knowledge storage” — and only 2/8 = 25% of that storage is accessed per token.
Mixtral’s routing is token-level, not sequence-level: the same word in different contexts can be routed to completely different experts. The router makes a fresh decision every token, every layer.
DeepSeek-V3 (671B total / 37B active) introduces two innovations beyond standard MoE:
1. Fine-Grained Expert Segmentation
Instead of 8 large experts, DeepSeek uses 256 small experts per layer with top-8 routing. Each expert is ~1/32 the size of a Mixtral expert. This gives the router much finer-grained control — you can build any “combination” of specializations rather than picking from 8 coarse buckets.
2. Shared Experts
DeepSeek designates a small number of experts as always-active shared experts. Every token passes through these, regardless of routing. The shared experts handle common, general-purpose transformations; the routed experts handle token-specific specialization.
DeepSeek MoE Layer:
token → [Shared Expert 1] ──────────────────────────────┐
→ [Shared Expert 2] ──────────────────────────────┤
→ Router → Top-K from 256 routed experts ─────────┴→ sum → output
This hybrid approach lets the model separate “what every token needs” from “what this specific token needs” — a cleaner decomposition than pure routing.
OpenAI has never confirmed GPT-4’s architecture. But the evidence is strong:
The economics make it inevitable: a 1.8T MoE with 2/16 active experts costs the same to run as a ~200B dense model, but has the knowledge capacity of a 1.8T one. At frontier scale, there’s no competitive reason to use dense.
MoE sounds like a free lunch. It isn’t. The hidden cost is all-to-all communication.
In distributed training, model parameters are split across devices (tensor parallelism) or layers (pipeline parallelism). With MoE, you add a third dimension: expert parallelism — each expert lives on a different device.
This creates a routing problem at the hardware level: after the router decides which expert each token goes to, you have to physically ship that token’s activations to the device that holds the relevant expert. After the expert processes it, the results ship back.
This all-to-all collective — where every device sends data to every other device — is expensive on current hardware. The NVLink bandwidth between GPUs is fast, but not free. At the scale of 256 experts across 32 nodes, this communication can dominate training time.
DeepSeek’s engineering contribution (described in their training paper) is a custom all-to-all kernel that overlaps communication with computation — hiding the latency by starting the next layer’s attention while the current layer’s expert results are still in transit. This is where the real engineering work happens in production MoE systems.
| Dense | MoE | |
|---|---|---|
| Training FLOPs per token | High | Low |
| Inference FLOPs per token | High | Low |
| Memory (weights) | Low | High |
| Training stability | High | Moderate (needs load balancing) |
| Serving complexity | Simple | Complex (expert parallelism) |
| Good for | Smaller models, simple serving | Frontier scale, research |
| Bad for | Frontier scale (too expensive) | Edge/mobile deployment |
The crossover point is roughly 30–70B parameters: below that, dense is simpler and sufficient; above that, MoE becomes the only economically viable path to higher capability.
MoE breaks the compute-parameter link — you can have 10× more parameters for the same per-token FLOPs, because only a fraction of experts activate per token.
The router is a learned function — it figures out which tokens need which specializations entirely from gradient descent, with no human-defined expert roles.
Load balancing is non-negotiable — without the auxiliary loss, expert collapse happens within the first few thousand steps.
Expert capacity introduces token dropping — a necessary engineering compromise for distributed training that the model learns to tolerate.
DeepSeek’s fine-grained + shared expert design is the current state of the art — more routing flexibility, cleaner separation of general vs. specialized computation.
The real cost is communication, not compute — building production MoE systems is a distributed systems problem as much as an ML problem.
Outrageously Large Neural Networks: The Sparsely-Gated Mixture-of-Experts Layer — Shazeer et al., Google Brain (2017) The paper that revived MoE for deep learning. Introduced the top-K gating mechanism and the load balancing auxiliary loss. Everything since builds on this.
Switch Transformers: Scaling to Trillion Parameter Models with Simple and Efficient Sparsity — Fedus et al., Google (2021) Simplified MoE to top-1 routing, showed you could scale to 1T+ parameters. First clear demonstration that MoE could match dense models at trillion-parameter scale.
GLaM: Efficient Scaling of Language Models with Mixture-of-Experts — Du et al., Google (2021) Trained a 1.2T MoE model using 1/3 the energy of GPT-3 training. Landmark result on MoE efficiency vs. dense models.
Mixtral of Experts — Jiang et al., Mistral AI (2024) The architecture paper behind Mixtral 8x7B and 8x22B. Clean, readable description of token-level routing with top-2 selection. The open-source MoE baseline.
DeepSeek-V3 Technical Report — DeepSeek AI (2024) 671B parameters, 37B active, trained for ~$5.5M. Describes fine-grained expert segmentation, shared experts, and the custom all-to-all communication kernel. State of the art in MoE efficiency.
DeepSeek-V2: A Strong, Economical, and Efficient Mixture-of-Experts Language Model — DeepSeek AI (2024) Predecessor to V3. Introduces the DeepSeekMoE architecture — the fine-grained + shared expert design — and validates it at scale.
ST-MoE: Designing Stable and Transferable Sparse Expert Models — Zoph et al., Google (2022) A detailed investigation into what makes MoE training unstable and how to fix it. Router z-loss, capacity factor tuning, and expert dropout. Essential reading before training your own MoE.
Towards Understanding Mixture of Experts in Deep Learning — Chen et al. (2022) Theoretical analysis of why MoE works: shows that sparse routing implicitly implements a form of conditional computation that dense models can’t replicate efficiently.
Expert Choice Routing: Better MoE with Expert-Selected Tokens — Zhou et al., Google (2022) Inverts the routing: instead of tokens choosing experts, experts choose tokens. Eliminates dropped tokens, achieves perfect load balance. An elegant alternative to the standard approach.
MoE is a bet on a simple idea: not every token needs the same computation. The remarkable thing is how much that bet pays off. A model that thinks selectively — routing each token to relevant specialists, ignoring the rest — turns out to be both more capable and cheaper to run than one that applies maximum force to everything. Intelligence, apparently, is less about brute force and more about knowing when not to think.
]]>Every company building on LLMs eventually hits the same wall. The model hallucinates facts. It doesn’t know about events after its training cutoff. You can’t fit your entire knowledge base into a context window. The answer everyone reaches for: Retrieval-Augmented Generation (RAG).
The naive version is simple: chunk your documents into paragraphs, embed them with a sentence transformer, store in a vector database, and at query time retrieve the top-K most similar chunks. Stuff them into the prompt. Done.
This works. Up to a point.
Then you discover that your retrieval misses the most relevant documents 30% of the time. That questions about relationships between entities fail completely. That the 500-token chunks you split on periods don’t align with how your model reasons. That users asking “what did the CEO say about layoffs last quarter?” get back chunks about layoffs from three years ago because the embeddings didn’t capture recency.
The gap between naive RAG and production RAG is wide. This post covers the techniques that close it: HyDE, reranking, ColBERT, and Graph RAG — with implementations you can actually use.
Before fixing things, name what’s broken:
1. Query-document mismatch Queries are short and abstract (“what causes transformer training instability?”). Documents are long and concrete (a paragraph describing gradient norms spiking due to attention softmax saturation). Their embeddings live in different regions of the vector space — not because the content is unrelated, but because the linguistic form is different.
2. Single-vector bottleneck A single embedding vector must compress an entire 512-token chunk. Complex chunks that discuss multiple concepts average their meaning into one point — and retrieval can fail on any of those concepts individually.
3. Semantic similarity ≠ relevance The most semantically similar chunk isn’t always the most useful one. A question about a drug’s side effects might return chunks that are topically related but don’t actually answer the question. Relevance requires understanding the intent of the query, not just its words.
4. Flat document structure Documents have structure — sections, subsections, tables, references between chapters. Chunking destroys this. A chunk about “revenue” in Q3 earnings doesn’t know it’s three paragraphs after the CFO’s statement that preceded it.
Each of these has a targeted fix.
Hypothetical Document Embeddings (HyDE) attacks the query-document mismatch directly. Instead of embedding the raw query, you ask an LLM to generate a hypothetical document that would answer the query — then embed that hypothetical document and use it for retrieval.
The intuition: a generated answer paragraph lives in the same linguistic register as real document chunks. Embedding it retrieves documents that look like answers, not documents that look like questions.
from openai import OpenAI
import numpy as np
client = OpenAI()
def hyde_retrieve(query: str, vector_store, top_k: int = 5):
"""
HyDE: generate a hypothetical answer, embed it, retrieve with that embedding.
"""
# Step 1: Generate a hypothetical document that answers the query
response = client.chat.completions.create(
model="gpt-4o-mini",
messages=[
{
"role": "system",
"content": (
"Write a short, factual paragraph (3-5 sentences) that directly "
"answers the following question. Write as if this were an excerpt "
"from a technical document. Do not mention that you are generating "
"a hypothetical answer."
),
},
{"role": "user", "content": query},
],
temperature=0.0,
)
hypothetical_doc = response.choices[0].message.content
# Step 2: Embed the hypothetical document (not the raw query)
embedding_response = client.embeddings.create(
model="text-embedding-3-small",
input=hypothetical_doc,
)
hyde_embedding = np.array(embedding_response.data[0].embedding)
# Step 3: Retrieve using the hypothetical document's embedding
results = vector_store.similarity_search_by_vector(hyde_embedding, k=top_k)
return results, hypothetical_doc
HyDE consistently improves retrieval recall on knowledge-intensive tasks, especially for technical or domain-specific queries where the vocabulary gap between question and answer is large. The cost is one extra LLM call per query — worth it for most production use cases.
When HyDE helps most: scientific literature search, legal document retrieval, technical documentation QA.
When it hurts: queries where the LLM generates a confidently wrong hypothetical. If the model hallucinates the hypothetical, you retrieve documents that support the hallucination. Always log both the query and the hypothetical for debugging.
Dense retrieval (embedding similarity) is fast but coarse. It finds roughly-relevant documents. A reranker is a cross-encoder that takes (query, document) as a pair and produces a precise relevance score — much slower than embedding similarity, but much more accurate.
The two-stage pipeline:
Query → Dense Retrieval (fast, recall-focused) → Top 50 candidates
↓
Reranker (slow, precision-focused)
↓
Top 5 reranked results → LLM
The reranker sees both query and document simultaneously — it can model their interaction, not just their individual semantics. This catches relevance signals that embedding similarity misses entirely.
from sentence_transformers import CrossEncoder
from typing import List, Tuple
class TwoStageRetriever:
def __init__(self, vector_store, reranker_model: str = "cross-encoder/ms-marco-MiniLM-L-6-v2"):
self.vector_store = vector_store
self.reranker = CrossEncoder(reranker_model)
def retrieve(self, query: str, initial_k: int = 50, final_k: int = 5) -> List[dict]:
# Stage 1: Dense retrieval — cast a wide net
candidates = self.vector_store.similarity_search(query, k=initial_k)
# Stage 2: Rerank with cross-encoder
pairs = [(query, doc.page_content) for doc in candidates]
scores = self.reranker.predict(pairs)
# Sort by reranker score, take top final_k
ranked = sorted(
zip(scores, candidates),
key=lambda x: x[0],
reverse=True
)
return [{"document": doc, "score": float(score)} for score, doc in ranked[:final_k]]
Reranker model options:
| Model | Size | Speed | Quality |
|---|---|---|---|
cross-encoder/ms-marco-MiniLM-L-6-v2 | 22M | Fast | Good baseline |
cross-encoder/ms-marco-MiniLM-L-12-v2 | 33M | Moderate | Better |
BAAI/bge-reranker-v2-m3 | 568M | Slow | Excellent |
| Cohere Rerank API | — | API call | Production-grade |
The initial_k / final_k ratio matters: retrieve 50, rerank to 5. If you retrieve too few initially, the reranker can’t save you — the relevant document was never in the candidate set. If you rerank too many, latency explodes. 30–100 → 3–10 is the typical production range.
HyDE and reranking are two ends of a spectrum: fast-and-coarse vs. slow-and-precise. ColBERT sits in the middle with a fundamentally different approach: late interaction.
In standard dense retrieval, you embed a query into one vector and documents into one vector, then compute dot product. The entire semantic content of the query is compressed into a single point.
ColBERT instead produces one embedding per token — both for the query and the document. Relevance is computed as the sum of maximum similarities between query tokens and document tokens:
Score(q, d) = Σ_{i ∈ query_tokens} max_{j ∈ doc_tokens} (q_i · d_j)
Each query token independently finds its best matching document token. A query like “transformer training instability” has its “instability” token match against “gradient explosion” in the document, and its “transformer” token match against “attention layer” — neither match alone would be sufficient, but their sum is.
import torch
import torch.nn.functional as F
from transformers import AutoTokenizer, AutoModel
class ColBERTRetriever:
def __init__(self, model_name: str = "colbert-ir/colbertv2.0"):
self.tokenizer = AutoTokenizer.from_pretrained(model_name)
self.model = AutoModel.from_pretrained(model_name)
self.model.eval()
def encode(self, texts: list[str], is_query: bool = False) -> torch.Tensor:
"""
Encode texts into per-token embeddings.
Returns: (batch, seq_len, dim)
"""
prefix = "[Q] " if is_query else "[D] "
prefixed = [prefix + t for t in texts]
tokens = self.tokenizer(
prefixed,
padding=True,
truncation=True,
max_length=512,
return_tensors="pt"
)
with torch.no_grad():
output = self.model(**tokens)
# L2-normalize each token embedding
embeddings = F.normalize(output.last_hidden_state, p=2, dim=-1)
return embeddings
def score(self, query_embs: torch.Tensor, doc_embs: torch.Tensor) -> torch.Tensor:
"""
Late interaction scoring: sum of max similarities.
query_embs: (q_len, dim)
doc_embs: (d_len, dim)
Returns: scalar relevance score
"""
# Similarity matrix: (q_len, d_len)
sim_matrix = torch.matmul(query_embs, doc_embs.T)
# Max over document tokens for each query token
max_sim = sim_matrix.max(dim=-1).values # (q_len,)
# Sum across query tokens
return max_sim.sum()
def retrieve(self, query: str, doc_embeddings: list, top_k: int = 5):
query_embs = self.encode([query], is_query=True)[0] # (q_len, dim)
scores = []
for doc_emb in doc_embeddings:
score = self.score(query_embs, doc_emb)
scores.append(score.item())
top_indices = sorted(range(len(scores)), key=lambda i: scores[i], reverse=True)[:top_k]
return [(i, scores[i]) for i in top_indices]
The scalability trick: document token embeddings are precomputed and stored (ColBERT indexes can be built with the RAGatouille library). At query time, you only compute query embeddings on the fly. The max-similarity operation over precomputed document embeddings is fast enough for millions of documents using PLAID (the ColBERT production indexing system).
When ColBERT shines: long documents where individual passages have multiple distinct topics, queries that require matching several specific concepts simultaneously, technical retrieval where exact terminology matters.
All three techniques above assume documents are independent, unrelated chunks. But real knowledge has structure: entities reference each other, concepts build on each other, facts have provenance. A knowledge base isn’t a bag of paragraphs — it’s a graph.
Graph RAG (popularized by Microsoft Research in 2024) builds an explicit knowledge graph from your documents and uses graph traversal alongside or instead of vector search.
The pipeline has two phases:
from openai import OpenAI
import networkx as nx
import json
client = OpenAI()
def extract_entities_and_relations(text: str) -> dict:
"""Use LLM to extract a structured knowledge graph from a text chunk."""
response = client.chat.completions.create(
model="gpt-4o",
messages=[
{
"role": "system",
"content": """Extract entities and relationships from the text.
Return JSON with this structure:
{
"entities": [{"name": str, "type": str, "description": str}],
"relations": [{"source": str, "relation": str, "target": str}]
}
Only include clearly stated facts. Be conservative."""
},
{"role": "user", "content": text}
],
response_format={"type": "json_object"},
temperature=0.0,
)
return json.loads(response.choices[0].message.content)
def build_knowledge_graph(chunks: list[str]) -> nx.DiGraph:
G = nx.DiGraph()
for chunk in chunks:
extracted = extract_entities_and_relations(chunk)
for entity in extracted.get("entities", []):
G.add_node(
entity["name"],
type=entity["type"],
description=entity["description"]
)
for rel in extracted.get("relations", []):
G.add_edge(
rel["source"],
rel["target"],
relation=rel["relation"]
)
return G
def graph_rag_retrieve(query: str, graph: nx.DiGraph, vector_store, top_k: int = 5):
"""
Two-path retrieval:
1. Vector search for relevant chunks
2. Entity extraction from query → graph traversal for related context
"""
# Path 1: Standard vector retrieval
vector_results = vector_store.similarity_search(query, k=top_k)
# Path 2: Extract entities from query, traverse the graph
entity_response = client.chat.completions.create(
model="gpt-4o-mini",
messages=[
{
"role": "system",
"content": "Extract entity names from this query. Return a JSON array of strings."
},
{"role": "user", "content": query}
],
response_format={"type": "json_object"},
temperature=0.0,
)
query_entities = json.loads(entity_response.choices[0].message.content).get("entities", [])
# Gather neighborhood context for each query entity
graph_context = []
for entity in query_entities:
if entity not in graph:
continue
# Direct neighbors (1-hop)
neighbors = list(graph.neighbors(entity)) + list(graph.predecessors(entity))
for neighbor in neighbors[:10]: # cap at 10 neighbors
edge_data = graph.get_edge_data(entity, neighbor) or graph.get_edge_data(neighbor, entity)
relation = edge_data.get("relation", "related to") if edge_data else "related to"
graph_context.append(f"{entity} {relation} {neighbor}")
# Include entity description
node_data = graph.nodes.get(entity, {})
if node_data.get("description"):
graph_context.append(f"{entity}: {node_data['description']}")
return {
"vector_results": vector_results,
"graph_context": graph_context,
"query_entities": query_entities,
}
Consider a corpus of earnings call transcripts. The question: “How has the relationship between Amazon’s AWS margins and its advertising revenue changed over the past three years?”
Vector retrieval returns chunks that mention AWS margins and chunks that mention advertising revenue — independently. It has no way to represent the relationship between these two things over time without explicit graph structure connecting them.
Graph RAG extracts AWS_Margins --[affects]--> Advertising_Investment, Advertising_Revenue --[grew_by]--> 27%_in_Q3_2024, and so on. Traversing from the query entities gives back the entire chain of relationships — exactly what the question is asking for.
Microsoft’s GraphRAG paper showed 3–4× improvement in “global question” tasks (questions requiring synthesis across multiple documents) compared to naive RAG. Local question performance was roughly equivalent. Graph construction is expensive (many LLM calls), but for static corpora, you pay it once.
These techniques compose. A production RAG system typically layers several:
class ProductionRAG:
def __init__(self, vector_store, graph, reranker_model):
self.vector_store = vector_store
self.graph = graph
self.reranker = CrossEncoder(reranker_model)
self.llm = OpenAI()
def retrieve(self, query: str) -> dict:
# 1. HyDE: generate hypothetical answer for better query embedding
hypothetical = self._generate_hypothetical(query)
# 2. Dual retrieval: embed both raw query and hypothetical, merge results
raw_results = self.vector_store.similarity_search(query, k=30)
hyde_results = self.vector_store.similarity_search(hypothetical, k=30)
# Deduplicate by document ID
seen = set()
candidates = []
for doc in raw_results + hyde_results:
if doc.metadata["id"] not in seen:
seen.add(doc.metadata["id"])
candidates.append(doc)
# 3. Graph augmentation: pull in relationship context
graph_context = self._graph_retrieve(query)
# 4. Rerank the merged candidate set
pairs = [(query, doc.page_content) for doc in candidates]
scores = self.reranker.predict(pairs)
ranked = sorted(zip(scores, candidates), key=lambda x: x[0], reverse=True)
top_docs = [doc for _, doc in ranked[:5]]
return {
"retrieved_docs": top_docs,
"graph_context": graph_context,
"hypothetical": hypothetical,
}
def answer(self, query: str) -> str:
context = self.retrieve(query)
doc_text = "\n\n---\n\n".join([d.page_content for d in context["retrieved_docs"]])
graph_text = "\n".join(context["graph_context"])
response = self.llm.chat.completions.create(
model="gpt-4o",
messages=[
{
"role": "system",
"content": (
"Answer the user's question using only the provided context. "
"If the context doesn't contain the answer, say so explicitly. "
"Cite sources by referencing document metadata where available."
),
},
{
"role": "user",
"content": (
f"Question: {query}\n\n"
f"Document Context:\n{doc_text}\n\n"
f"Relationship Context:\n{graph_text}"
),
},
],
)
return response.choices[0].message.content
def _generate_hypothetical(self, query: str) -> str:
response = self.llm.chat.completions.create(
model="gpt-4o-mini",
messages=[
{"role": "system", "content": "Write a 3-sentence factual answer to this question."},
{"role": "user", "content": query},
],
temperature=0.0,
)
return response.choices[0].message.content
def _graph_retrieve(self, query: str) -> list[str]:
# Entity extraction + graph traversal (as shown above)
...
Advanced retrieval can’t compensate for bad chunks. A few patterns worth knowing:
Semantic chunking — instead of fixed-size splits, chunk at semantic boundaries detected by embedding distance between consecutive sentences. When the embedding distance spikes, you’ve crossed a topic boundary.
from sentence_transformers import SentenceTransformer
import numpy as np
def semantic_chunk(text: str, threshold: float = 0.3) -> list[str]:
model = SentenceTransformer("all-MiniLM-L6-v2")
sentences = text.split(". ")
embeddings = model.encode(sentences)
# Cosine distance between consecutive sentences
chunks, current_chunk = [], [sentences[0]]
for i in range(1, len(sentences)):
cos_sim = np.dot(embeddings[i-1], embeddings[i]) / (
np.linalg.norm(embeddings[i-1]) * np.linalg.norm(embeddings[i])
)
if 1 - cos_sim > threshold: # topic boundary detected
chunks.append(". ".join(current_chunk))
current_chunk = [sentences[i]]
else:
current_chunk.append(sentences[i])
chunks.append(". ".join(current_chunk))
return chunks
Parent-child chunking — index small child chunks (128 tokens) for retrieval precision, but return the larger parent chunk (512 tokens) to the LLM for context. You get the best of both: precise matching, rich context.
Proposition indexing — extract atomic factual statements from each chunk, embed and index those instead of the raw text. Each proposition is a single retrievable claim. Much higher precision for fact-seeking queries; higher construction cost.
Query-document mismatch is the root cause of most retrieval failures — HyDE fixes it by generating a hypothetical answer that lives in the same linguistic space as your documents.
Dense retrieval is for recall; reranking is for precision — retrieve 50, rerank to 5. Never use a single-stage retriever in production.
ColBERT’s per-token embeddings capture multi-concept queries that single-vector retrieval compresses away. Worth the indexing overhead for complex domains.
Graph RAG isn’t about replacing vector search — it’s about adding structure that vector search is blind to. Use it when your queries ask about relationships, not just topics.
Chunking strategy is the foundation everything else sits on — semantic chunking and parent-child indexing are the minimum viable improvements over naive fixed-size splits.
These techniques compose — HyDE + two-stage reranking + graph augmentation is a coherent production stack, not competing approaches.
Retrieval-Augmented Generation for Knowledge-Intensive NLP Tasks — Lewis et al., Facebook AI (2020) The original RAG paper. Combines a dense retriever (DPR) with a seq2seq generator (BART). Sets the vocabulary and evaluation benchmarks the field still uses.
Dense Passage Retrieval for Open-Domain Question Answering (DPR) — Karpukhin et al., Facebook AI (2020) The bi-encoder retrieval model that underpins most dense retrieval systems. Training with in-batch negatives is the key technique.
MS MARCO: A Human Generated Machine Reading Comprehension Dataset — Bajaj et al., Microsoft (2016) The dataset most reranker models are trained on. Understanding its structure helps you evaluate whether a reranker will generalize to your domain.
A Setwise Approach for Effective and Highly Efficient Zero-shot Ranking with Large Language Models — Zhuang et al. (2023) LLM-based reranking — using GPT-4 or similar as a reranker directly. Slower but often better than cross-encoders for out-of-domain queries.
ColBERT: Efficient and Effective Passage Search via Contextualized Late Interaction over BERT — Khattab & Zaharia, Stanford (2020) The original ColBERT paper. Introduces late interaction and the deferred MaxSim operation that makes it scalable.
ColBERTv2: Effective and Efficient Retrieval via Lightweight Late Interaction — Santhanam et al., Stanford (2021) Improves ColBERT with residual compression — reduces storage 6–10× while maintaining quality. The version used in production systems.
RAGatouille — Benjamin Clavié The library that makes ColBERT accessible in 10 lines of code. Wraps PLAID indexing, supports Hugging Face datasets, integrates with LangChain/LlamaIndex.
From Local to Global: A Graph RAG Approach to Query-Focused Summarization — Edge et al., Microsoft Research (2024) The GraphRAG paper. Shows 3–4× improvement on global synthesis tasks. Introduces community detection over the knowledge graph as a summarization strategy — compelling results on large corpora.
KGRAG: Knowledge Graph Enhanced Retrieval-Augmented Generation — (2023) Alternative approach using existing knowledge graphs (Wikidata, domain KGs) rather than LLM-extracted ones. Lower construction cost, less flexible.
Propositions as the Unit of Retrieval (Dense X Retrieval) — Chen et al. (2023) Atomic proposition indexing — extract and index individual factual claims rather than passages. Significant precision improvement for fact-seeking queries.
RAPTOR: Recursive Abstractive Processing for Tree-Organized Retrieval — Sarthi et al., Stanford (2024) Builds a hierarchical tree of summaries over your document corpus. Retrieval happens at multiple levels of abstraction simultaneously — strong on multi-hop reasoning tasks.
Naive RAG is a prototype. Production RAG is a system. The difference isn’t any single technique — it’s building a retrieval stack that thinks about where each failure mode comes from and applies the right fix at the right layer. Get that right, and your LLM stops hallucinating not because you made it smarter, but because you stopped giving it reasons to guess.
]]>On April 7, 2026, Anthropic announced something unusual: a new frontier model that they have no intention of making generally available.
Claude Mythos Preview is Anthropic’s most capable model to date — outperforming Claude Opus 4.6 on every major coding and reasoning benchmark by significant margins. It can autonomously find zero-day vulnerabilities that survived decades of human review and millions of automated security tests. It can chain multiple Linux kernel vulnerabilities for privilege escalation without any human guidance.
And you cannot use it. At least not unless you’re one of the ~40+ organizations hand-picked to access it through Project Glasswing — a coalition of the largest technology companies in the world, convened specifically to give defenders an advantage over attackers before a model like this falls into the wrong hands.
This post covers what Mythos is, what it can do, how it compares to other Claude models, and what Anthropic’s decision to restrict it tells us about the future of frontier AI.
Claude Mythos Preview is a cybersecurity-specialized frontier model — Anthropic’s most capable AI system ever released (even as a preview), designed from the ground up to assist with defensive security workflows.
The word “preview” is doing real work here. This is not a generally available product. It is a research preview offered to a curated set of organizations under strict access controls, as part of a deliberate strategy to arm defenders before attackers gain access to equivalent capabilities.
It is available via the Claude API, Amazon Bedrock, Google Cloud Vertex AI, and Microsoft Foundry — but only for invited organizations. There is no self-serve sign-up. Access requires application to the Cyber Verification Program or membership in Project Glasswing.
Pricing (for authorized users): $25 per million input tokens / $125 per million output tokens — roughly 5x the cost of Claude Opus 4.6, reflecting both the model’s capability level and the intentionally limited rollout.
To understand Mythos, you first need to understand why Anthropic announced it the way they did.
Project Glasswing (announced April 7, 2026) is a coalition that brings together:
AWS, Anthropic, Apple, Broadcom, Cisco, CrowdStrike, Google, JPMorganChase, the Linux Foundation, Microsoft, NVIDIA, and Palo Alto Networks
The explicit goal: ensure that AI’s most powerful security capabilities reach defenders before attackers. Anthropic describes it as “an urgent attempt” — the language is deliberate. They believe a model capable of what Mythos can do is coming regardless of whether Anthropic builds it. The question is who gets it first.
Commitments made at launch:
This is not a product launch. It is a coordinated deployment strategy built around a security thesis: defenders need this more urgently than the public needs access to it.
The headline capabilities are genuinely remarkable — and in some cases, should give you pause.
Mythos can find vulnerabilities that human reviewers and automated tools have missed — for years, sometimes decades.
Documented examples from Anthropic’s announcement:
A 27-year-old vulnerability in OpenBSD — a security-focused operating system that has been under continuous expert review since 1999. Mythos found a flaw that had survived nearly three decades of scrutiny.
A 16-year-old flaw in FFmpeg — the ubiquitous open-source multimedia framework used in virtually every video streaming platform, browser, and media application on the planet. This vulnerability survived 5 million automated test runs by fuzzing tools specifically designed to find bugs like it.
Linux kernel privilege escalation — Mythos autonomously identified and chained multiple separate vulnerabilities in the Linux kernel to achieve privilege escalation, without any human guidance or steering at any step.
The phrase “entirely autonomously, without any human steering” appears in Anthropic’s own description. This is not a model that helps a human security researcher — it is a model that is the security researcher, working independently.
Mythos operates as a full agentic system — it doesn’t just identify vulnerabilities, it can develop exploits, write and execute code, navigate file systems, and chain tool calls across long autonomous sessions. This puts it in a different capability class from models that primarily generate text for human review.
Beyond cybersecurity, Mythos is simply a more capable general reasoner than anything Anthropic has previously released. The benchmark numbers confirm this.
| Benchmark | Mythos Preview | Claude Opus 4.6 | Delta |
|---|---|---|---|
| CyberGym (Vulnerability Reproduction) | 83.1% | 66.6% | +16.5 pts |
| SWE-bench Pro (Real-world coding) | 77.8% | 53.4% | +24.4 pts |
| SWE-bench Verified | 93.9% | 80.8% | +13.1 pts |
| Terminal-Bench 2.0 | 82.0% | 65.4% | +16.6 pts |
| GPQA Diamond (Graduate reasoning) | 94.6% | 91.3% | +3.3 pts |
A few things stand out:
SWE-bench Pro is the most striking. This benchmark uses real-world GitHub issues from production codebases — not curated problems. Mythos resolves 77.8% of them. Opus 4.6 resolves 53.4%. That 24-point gap is enormous in practical terms — it means Mythos solves roughly half again as many real engineering problems as the best publicly available model.
GPQA Diamond shows the narrowest gap. Graduate-level science reasoning (physics, chemistry, biology) is the one domain where Mythos’ lead over Opus 4.6 shrinks to ~3 points. Both models are operating near the ceiling of this benchmark. The differentiation is overwhelmingly in agentic, tool-use, and code-execution tasks.
CyberGym is a new benchmark. It measures vulnerability reproduction — given a description of a known CVE, can the model reproduce the exploit? At 83.1%, Mythos is far ahead of anything previously available. This is not a benchmark most models even attempt.
Claude currently has three publicly available tiers, plus Mythos as a restricted preview:
| Model | Context | Pricing (in/out per MTok) | Best For |
|---|---|---|---|
| Claude Haiku 4.5 | 200k tokens | $1 / $5 | Speed, high-volume tasks |
| Claude Sonnet 4.6 | 1M tokens | $3 / $15 | Balanced speed + intelligence |
| Claude Opus 4.6 | 1M tokens | $5 / $25 | Complex reasoning, agentic tasks |
| Claude Mythos Preview | TBD | $25 / $125 | Cybersecurity, frontier capability |
Mythos sits at 5x the price of Opus 4.6 — and based on the benchmark numbers, earns it for the right workloads.
This is the part that makes Mythos genuinely interesting from an AI development perspective.
Anthropic has built a model that is meaningfully more capable than anything publicly available. Rather than releasing it broadly — which would maximize revenue and reach — they’ve chosen a controlled deployment specifically because of what the model can do.
Their reasoning, as stated:
Offense/defense asymmetry: A model that can find zero-days is dangerous in attacker hands and invaluable in defender hands. Releasing to defenders first, under access controls, tries to tilt that asymmetry.
Safeguards are still in development: Anthropic explicitly states they are “developing cybersecurity (and other) safeguards that detect and block the model’s most dangerous outputs” — and that these safeguards will be incorporated into future public Claude Opus releases. Mythos Preview ships without the full safety stack because the safety stack isn’t ready yet.
Verification before access: The Cyber Verification Program requires applicants to demonstrate legitimate security work. This isn’t perfect — verification can fail — but it’s a meaningful friction point that general API access wouldn’t provide.
This approach is a departure from the default “release and iterate” model that most AI labs follow. It raises genuinely hard questions: who decides which organizations are legitimate defenders? What happens when a Glasswing partner is compromised? How long before equivalent capabilities are available elsewhere?
Anthropic isn’t pretending these questions have clean answers. The framing of “urgent attempt” suggests they believe the window for this strategy is limited.
For security teams: The gap between AI-assisted and unassisted security work is about to widen dramatically. Teams with access to Mythos-class models will find vulnerabilities faster, assess attack surfaces more completely, and respond to incidents with better intelligence than teams without. This is not a marginal advantage.
For open source: The $4M in credits to open-source security organizations is significant. Projects like OpenBSD, the Linux kernel, curl, OpenSSL — the infrastructure everything else runs on — will have access to a model that has already demonstrated it can find bugs their own maintainers missed.
For AI development norms: Project Glasswing is an experiment in whether a controlled, coalition-based deployment can work for a dual-use capability. If it does — if defenders meaningfully get ahead of attackers because of it — it will influence how future frontier models with dangerous capabilities are released. If it fails, it will inform what stronger interventions might look like.
For developers: Mythos will eventually influence what’s in public Claude models. Anthropic’s statement that the cybersecurity safeguards will be incorporated into future Opus releases suggests a pipeline: build capability at the frontier, develop safety measures, then push both into general availability. The public models you use today are yesterday’s frontier models with better guardrails.
There are two paths:
1. Project Glasswing Partnership If your organization works on critical infrastructure security and could benefit from Glasswing partnership, the announcement page at anthropic.com/glasswing is the starting point. This is designed for large organizations — the current partners are some of the largest technology and security companies in the world.
2. Cyber Verification Program For individual security researchers and smaller organizations doing legitimate defensive security work, the Cyber Verification Program offers a path to expanded access to Mythos capabilities. This is the relevant path for penetration testers, vulnerability researchers, and security engineers.
Anthropic has not announced a timeline for general availability — and given the framing, it seems unlikely Mythos Preview will ever be generally available in its current form. The more likely path:
This mirrors how the rest of the Claude family has evolved: capabilities that required frontier models a year ago are now standard in Sonnet and Haiku. The frontier keeps moving. Mythos is today’s frontier.
Claude Mythos is the clearest signal yet that frontier AI capabilities are outpacing the safety infrastructure needed to deploy them broadly. Anthropic’s choice to restrict it rather than release it is either responsible restraint or an elaborate delay — probably both. Either way, the gap between what the most capable AI can do and what the public has access to just got measurably wider.
]]>As of April 7, 2026, the Artemis II crew is approximately Day 6 of their mission — returning from the Moon after a historic lunar flyby. This post covers the full story from preparation to present and what happens after splashdown.
On April 1, 2026, at 22:35 UTC, a 322-foot Space Launch System rocket lit up the night sky above Kennedy Space Center’s Launch Complex 39B. Aboard the Orion capsule: four astronauts about to become the first humans to travel beyond low Earth orbit in over 53 years.
This is Artemis II. Not a landing — a proving flight. A 10-day journey around the Moon and back, testing every system that will one day carry humans to the lunar surface, and eventually, to Mars.
Artemis II carries four astronauts, each making history in their own right.
A former Navy test pilot and veteran of the ISS, Wiseman was selected as Artemis II commander in 2023. At 58, he becomes the oldest person to travel beyond low Earth orbit. He previously served as Chief of the Astronaut Office at NASA. His role: ultimate responsibility for crew safety, mission decisions, and vehicle command.
A Navy Captain and fighter pilot with over 3,000 flight hours, Glover flew to the ISS on the first operational Crew Dragon mission in 2020. On Artemis II he becomes the first person of color to travel around the Moon — a milestone with enormous cultural significance 54 years after Apollo’s final mission.
Koch holds the record for the longest single spaceflight by a woman (328 days on the ISS). She is the first woman ever to travel to the Moon — a barrier that stood for the entire era of human spaceflight. Her expertise in life sciences and EVA operations makes her central to crew health monitoring during the mission.
For Hansen, this is his first spaceflight — and what a first flight. He becomes the first non-American to travel around the Moon, reflecting Canada’s deep partnership with NASA through the Lunar Gateway program and Canadarm3. The CSA contributed critical hardware to both the SLS and Orion programs.
Many people ask: why go around the Moon without landing? The answer is engineering discipline.
Artemis I (November 2022) flew the same free-return trajectory with an uncrewed Orion. It was a success by every measure — but post-flight inspection revealed unexpected heat shield erosion during reentry. Charred material had ablated in ways the models didn’t predict, raising questions about the capsule’s safety margins for a crewed return.
NASA convened an independent review team. After months of analysis, they concluded: the heat shield is safe, but we’ll fly a slightly steeper reentry angle to reduce the ablation issue. Artemis II became the test of that fix, with a real crew, under real conditions.
This is how spaceflight is done right. You don’t skip steps. You don’t rush the crewed landing because the public is impatient. You send four people around the Moon first, get them back safely, and then you land.
The path from Artemis I’s splashdown to Artemis II’s launch took over three years. Here is what happened:
After reaching a 23.5-hour elliptical high Earth orbit, the crew began the most critical early phase: verifying every Orion system works in space before committing to the Moon.
Life support, propulsion, navigation, communications, suit systems, displays — all checked methodically against pre-planned procedures. If anything had failed here, the crew could have returned to Earth within hours.
The highlight of Day 1: proximity operations with the spent ICPS upper stage. Wiseman manually flew Orion to within close range of the inert booster over a 70-minute demonstration — rehearsing the docking skills future Artemis missions will use with the Human Landing System.
The ICPS fired one final time: a 5-minute, 49-second burn that accelerated Orion from ~17,000 mph to ~24,500 mph — escape velocity from Earth orbit, aimed at the Moon.
This burn placed the crew on a free-return trajectory: a precisely calculated arc that loops around the Moon and returns to Earth using only gravity, requiring no additional propulsion for the return. If the main engine failed tomorrow, the crew would still come home. This passive safety margin was a deliberate design choice absent from Apollo missions.
Image credit: NASA. An Artemis II crew member gazes at Earth through Orion’s window during the outbound transit to the Moon.
Three days of cruise, system monitoring, and crew activities. The crew:
Image credit: NASA. The Orion spacecraft with the Moon visible in the distance — captured during Artemis II’s deep space transit.
The emotional and scientific peak of the mission.
Closest approach: approximately 4,067 miles (6,545 km) from the lunar far side. Not as close as a landing trajectory, but far beyond anything since Apollo — and intentionally so. The free-return trajectory naturally swings farther out than a powered lunar orbit insertion would.
Records broken at closest approach:
For 40 minutes, Orion passed behind the Moon, cutting off all radio contact with Earth — the same blackout Apollo crews experienced, now repeated for the first time in five decades.
The crew also witnessed a 57-minute solar eclipse as the Moon blocked the Sun — something no human had seen since the Apollo era.
Orion now coasts home on its free-return arc. Two trajectory correction burns fine-tune the reentry corridor. The crew monitors all systems, completes science objectives, and prepares for the most demanding phase: reentry.
Orion will re-enter Earth’s atmosphere at approximately 25,000 miles per hour — the fastest crewed reentry in history, faster even than Apollo returns because of Orion’s trajectory.
Unlike Artemis I, which used a skip reentry (bouncing off the upper atmosphere to shed speed gradually), Artemis II uses a direct steeper-angle reentry — the modification NASA made to address the heat shield erosion concern. This is the primary engineering validation of the entire mission.
Planned splashdown: April 10–11, 2026, Pacific Ocean near San Diego. USS San Diego is positioned for recovery, alongside NASA’s recovery team and medical personnel.
Artemis II is a milestone, not a destination. Here is what the program looks like beyond splashdown:
The first crewed landing on the Moon since Apollo 17’s Gene Cernan in December 1972. The mission will land near the lunar south pole — a region Apollo never visited, chosen because permanently shadowed craters there likely contain water ice, a critical resource for long-term exploration and potential fuel production.
Landing system: SpaceX Starship Human Landing System (HLS) — a lunar variant of Starship that will carry 2 of the 4 Orion crew members from lunar orbit to the surface and back. The other 2 remain in Orion.
Surface activities: 2 EVAs of approximately 2 hours each. The crew will collect samples, deploy instruments, and scout the terrain that future Artemis missions will inhabit for weeks.
Artemis IV introduces the Lunar Gateway — a small space station in lunar orbit that will serve as a staging point for surface operations. Artemis IV will deliver the first habitation module and begin building humanity’s permanent infrastructure beyond Earth orbit.
The Gateway is a fundamentally different approach from Apollo: instead of point-to-point missions, each Artemis mission will build infrastructure that the next one uses.
Later missions extend surface stays from hours to weeks. Pressurized rovers. Permanent habitats. Resource extraction experiments. The south pole becomes a research base, not just a landing site.
NASA’s stated long-term goal is Mars — and the Moon is the proving ground. Every system being tested on Artemis (life support for long duration, radiation mitigation, in-situ resource utilization) is directly applicable to a multi-year Mars transit. Artemis is not nostalgia for Apollo. It is the first phase of interplanetary civilization.
It is easy to process Artemis II as a news event — a launch, some records, a splashdown — and move on. But the significance is larger.
Generational continuity: Everyone alive during Apollo who followed the space program closely is now in their 70s or older. For the majority of humans alive today, no one has ever left low Earth orbit in their lifetime. Artemis II is the first time in most people’s lives that they can look up and know: right now, there are people near the Moon.
Diversity: Christina Koch is the first woman to travel to the Moon. Victor Glover is the first person of color. Jeremy Hansen is the first non-American. These aren’t symbolic gestures — they are the natural result of a broader talent pool and a more inclusive astronaut corps. The faces of exploration are changing.
International partnership: Artemis is not an American program with international contributors. It is a genuinely multinational effort. Canada, ESA, JAXA, and others contribute hardware, crew, and funding. The Lunar Gateway will be operated by multiple space agencies. This is the model for sustainable deep space exploration.
Commercial participation: SpaceX builds the landing system. Blue Origin builds the Gateway’s propulsion module. The era of government-only spaceflight is over. Artemis is proof that the public-private model can execute missions as ambitious as anything Apollo attempted.
All images in this post are courtesy of NASA and the Canadian Space Agency (CSA). NASA imagery is in the public domain. Images sourced from the NASA Image and Video Library.
To explore more Artemis imagery, visit:
The Moon is 238,855 miles away. Right now, four people are closer to it than any human has been since 1972. Follow along at nasa.gov.
]]>When the Claude Code source leaked, the AI developer community got an unexpected gift: a detailed look at how a production-grade agentic coding assistant is actually built under the hood. Not the marketing pitch — the real scaffolding. The tool loop. The prompt structure. The stream-parse-execute cycle that makes it feel like a colleague rather than a chatbot.
I read every line. Then I built my own version.
Forge is an open-source AI coding agent CLI written in Go. It does what Claude Code does — code review, commit generation, interactive AI chat, file editing, bash execution — but it’s a single 16MB binary that starts in under 50ms, runs fully offline via Ollama, and costs nothing if you don’t want to pay for an API.
This post is about what I learned building it and why Go was the right choice.
The core of Claude Code — and every serious agentic coding tool — is deceptively simple. It’s a loop:
1. Send context + tools to the model
2. Stream the response
3. If the model calls a tool → execute it, append result
4. Repeat until the model stops calling tools
5. Return final response
That’s it. The magic is entirely in:
The leak confirmed what interpretability researchers already knew: the “intelligence” in these tools is overwhelmingly in the underlying model. The scaffolding is plumbing. Good plumbing matters — but it’s learnable, replicable, and in Go, very fast.
Claude Code is TypeScript/Node. Most AI tooling is Python. I chose Go for three reasons:
1. Single binary deployment go build produces one self-contained executable. No node_modules, no virtualenv, no runtime dependency hell. You download Forge, you run Forge. That’s the entire installation.
2. Startup time Node.js cold-start on a typical dev machine: 200–400ms. Go: under 50ms. When you’re running forge review --staged as a pre-commit hook dozens of times a day, that difference compounds into real friction.
3. Goroutines for streaming AI responses stream as server-sent events. Go’s concurrency model — goroutines + channels — makes reading a stream while processing tool calls genuinely elegant. No async/await callback soup.
Forge has four main layers:
type Provider interface {
Chat(ctx context.Context, messages []Message, tools []Tool) (<-chan Event, error)
Models(ctx context.Context) ([]Model, error)
}
Both AnthropicProvider and OllamaProvider implement this. Swap the provider, everything else stays the same. This is what makes offline mode work — Ollama speaks a compatible tool-use format, so the agentic engine doesn’t know or care which backend it’s talking to.
The send-stream-tools-repeat loop from above, implemented cleanly:
func (a *Agent) Run(ctx context.Context, task string) error {
messages := []Message{{Role: "user", Content: task}}
for {
events, err := a.provider.Chat(ctx, messages, a.tools)
// ... stream events
if len(toolCalls) == 0 {
break // model is done
}
for _, call := range toolCalls {
result := a.executeTool(ctx, call)
messages = append(messages, toolResultMessage(call.ID, result))
}
}
return nil
}
| Tool | What it does |
|---|---|
bash | Execute shell commands |
read_file | Read file contents |
write_file | Create or overwrite files |
edit_file | Targeted string replacement (preserves context) |
grep | Regex search across files |
glob | File pattern matching |
web_fetch | Fetch and summarize web pages |
git_diff | Get staged/unstaged diffs |
list_dir | Directory listing |
todo_write | Manage task lists |
| + 6 more | Notebook, task tracking, etc. |
This is essentially the same tool surface as Claude Code. The model uses them identically because they’re described the same way in the system prompt.
The interactive mode uses Bubble Tea — a Go framework for terminal UIs following the Elm architecture. It handles the chat interface, streaming output rendering, and keyboard shortcuts cleanly without any JavaScript-framework complexity.
Building this taught me what the scaffold does and doesn’t do.
The biggest practical challenge is context window management. As a conversation grows, you hit the model’s token limit. The naive fix — truncate old messages — breaks tool-call continuity. The right approach is selective summarization: keep tool results recent, summarize earlier conversation turns, never truncate mid-tool-call.
Getting this wrong means the model loses track of what files it’s already edited. Getting it right means long multi-file refactors work coherently.
I spent more time on the system prompt than on any other single component. The model’s “personality” as a coding agent — how it plans before acting, whether it reads files before editing them, how it handles errors — is entirely encoded in that prompt. The code is just infrastructure for delivering it.
# Download binary (macOS Apple Silicon)
curl -L https://github.com/sai-sridhar-repo-07/tarra-claw/releases/latest/download/forge-darwin-arm64 -o forge
chmod +x forge && sudo mv forge /usr/local/bin/
# Set your key
export ANTHROPIC_API_KEY=sk-ant-...
# Review your staged changes before committing
forge review --staged
# Install Ollama
brew install ollama && ollama serve &
# Pull a coding model (4GB, worth it)
ollama pull qwen2.5-coder:7b
# Run Forge against it — no API key needed
FORGE_PROVIDER=ollama forge review --staged
| Model | Size | Best for |
|---|---|---|
llama3.2 | 2GB | General chat, quick tasks |
qwen2.5-coder:7b | 4GB | Code review, editing (recommended) |
deepseek-coder-v2 | 8GB | Large codebases, complex refactors |
forge review --staged — Dedicated code review command. Analyzes your git diff before you commit and flags real issues: divide-by-zero bugs, unhandled exceptions, SQL injection risks, logic errors. Not a style linter. Actual bug detection.
forge commit — Generates a conventional commit message from your staged diff. Reads your actual changes, not a template. Saves the 30 seconds of message-writing on every commit.
forge review --branch main — Review the full diff between your branch and main. Useful before opening a PR.
Zero lock-in — Your data never leaves your machine in Ollama mode. No telemetry, no accounts, MIT licensed.
Drop a ~/.config/forge/config.yaml if you want persistent settings:
provider: ollama # or "anthropic"
ollama_model: qwen2.5-coder:7b
max_tokens: 8096
auto_approve: false # set true to skip tool-execution confirmations
Or use environment variables for one-off overrides:
FORGE_PROVIDER=anthropic ANTHROPIC_API_KEY=sk-ant-... forge run "refactor the auth module to use JWT"
A few things I deliberately left out of v1 that are on the roadmap:
The leak wasn’t really about Claude Code. It was a reminder that agentic AI tools — however impressive they feel — are thin scaffolding around a powerful model. The scaffolding matters, but it’s not magic. It’s software. It can be studied, replicated, and improved.
If you’re a developer who’s been treating these tools as black boxes, I’d encourage you to build one. Even a minimal implementation clarifies how they work in a way that no amount of using them ever will.
Forge is MIT licensed. Read it, fork it, break it, make it better.
github.com/sai-sridhar-repo-07/tarra-claw
kubectl, hugo, gh, and now Forge.forge commit output. Enables automated changelogs and semantic versioning.forge review --staged for automated AI review on every commit.