Predicting in latent space

Most of what we build at Natural is software. But underneath it is a belief that today's models, however impressive, aren't on the road to systems that keep learning, remember, and reason the way people do. This is the first in a series of notes on the research we're doing about that. It's an introduction, not a report — we'll publish results as we get them.

The bet

A standard language model is trained to guess the next token. Every word, every piece of punctuation. It's a remarkably productive objective, but it asks the model to be good at reproducing the surface of text, and it isn't obvious that surface reproduction is what produces understanding.

The alternative we're exploring is to have the model predict in its own representation space. Rather than "what's the next word", the question becomes "given what I've read so far, what will the next span of text be about?" — answered not in words but as a compact latent state the model has learned to produce. Words come later, from a separate readout head that decodes a latent back into text.

This is the JEPA idea — joint-embedding predictive architectures, from LeCun and colleagues — applied to language. There's a small but growing body of theory suggesting that models which learn to predict their own latents can pick up the structure of their data from far fewer examples than token predictors, because they're free to ignore the unpredictable detail and keep only what carries forward. That sample-efficiency claim has been shown on synthetic data. Nobody has shown it cleanly on real text yet. That's the gap we're aiming at.

What we're building

Phase 1 is deliberately small: nanoGPT scale, single GPU, text only. There are three trained pieces:

• A text encoder that reads a span (roughly a sentence) and pools it into one latent vector.
• A predictive trunk — a small causal transformer that operates purely over the sequence of latents and predicts the next one. It never sees tokens.
• A text decoder that reconstructs a span from its latent, so latents stay grounded in real content and we have a way to actually emit text.

Alongside it we train a matched next-token baseline, same parameter count, so every number we report is a pair: latent model versus token model. We're also building a two-level variant, where a second encoder pools several span latents into a paragraph-level one, to test whether explicit hierarchy earns its keep or whether a flat predictor discovers it on its own.

The trunk is the important boundary. Everything text-specific lives in the encoder and decoder; the trunk knows nothing about vocabularies or sentences. Text is the pilot modality, not the destination.

The catch

Token prediction has a free, uncheatable training signal: the real next word. Latent prediction doesn't. The target the model is trying to hit is produced by its own encoder — so the easiest way to "win" is for the encoder to output the same vector for everything, and the predictor to predict it. Loss goes to zero. Nothing has been learned.

The whole project is: build this, measure collapse honestly, and compare to a token baseline.

That failure is called representation collapse, and it comes in two flavours — the encoder outputs near-constant vectors, or it quietly confines them to a thin subspace so the loss looks fine while the representation is empty. The second one is the subtle one, and the one we expect to fight on text.

So a large part of Phase 1 is instrumentation. We log the effective rank of the latents, their per-dimension variance, and pairwise similarity throughout training, and we're implementing three different anti-collapse mechanisms (VICReg-style variance and covariance penalties, a contrastive term, and an EMA target encoder) so we can say which of them holds the representation open — and, importantly, whether the version with no regulariser collapses. We expect it to. That's data.

What would count

The headline result is a learning curve. Train both models across training-set sizes from a thousand examples up to a million, plot downstream performance against data, and look at the shape. If the latent model's curve sits to the left of the token baseline's — reaching the same performance with fewer examples, converging as data grows — the theory's sample-efficiency prediction has survived contact with natural language. If it doesn't, we've located where the theory's assumptions break down for real text.

Either is a real result. This is not an attempt to build a frontier model. A clean negative — "latent text prediction collapses unless X, and underperforms tokens by Y" — is a perfectly good thing to publish, and we'll publish it if that's what we find.

Where this goes

If it works, a latent trunk is a foundation for the things we actually care about. Because the trunk is modality-agnostic, image and audio adapters can be swapped in without touching it. Because prediction error in latent space is surprise, a memory system that writes when the model is surprised falls out almost for free. And because the unit of state is a latent rather than a token, there's a path to a genuinely stateful model, one where memory and context stop being separate objects.

None of that is being built now. Phase 1 is one question, asked carefully. We'll write again when we have numbers.