World Models: Learning to Predict, Not Just Describe

Ask a language model what happens if you remove a load-bearing step from a process, and it will give you a fluent, plausible answer drawn from text it has seen. Ask it to be reliably right across a thousand variations of that question, and the cracks show. It describes the world well. It doesn't model it.

A world model is the other thing: a model that learns the dynamics of a system, how state evolves over time and under intervention, and can predict what comes next. That is the difference between a system that can narrate a factory floor and one that can anticipate when a line is about to fail.

Predict the next state, not the next token

Large language models are trained to predict the next token. It's a brilliant objective for language, but it optimizes for describing sequences rather than understanding the system that generated them.

World models change the objective. Instead of learning what word comes next, they learn what state comes next. And, crucially, they learn it in a learned representation space, not in raw observations.

Predicting every pixel of the future is a waste. Most of that detail is irrelevant or plain unpredictable. What you actually want is a prediction of the parts that matter, in a representation that has already thrown away the parts that don't. This is the core insight behind joint-embedding predictive architectures (the JEPA line of research): rather than reconstructing the future in full detail, predict its representation. Generative models burn enormous capacity reproducing noise. Predictive-embedding models spend that capacity on structure, and for most decision-making tasks structure is all you need.

Why "in representation" is the whole point

There are a few ways to model a future you can't fully observe. You can generate it outright, in pixels or tokens, which is expensive and spends most of its effort on detail nobody needs. You can predict raw signals, which tends to be brittle because the world is noisy and high-dimensional. Or you can predict a learned representation, which holds up far better because the model itself decides what is worth encoding.

The third option is where the field is converging. A world model that predicts in latent space can also be trained largely self-supervised, from unlabeled sequences of how things actually behave. Operational systems produce exactly that kind of data in abundance, and no one has time to label it.

Where it changes what AI can do

World models are no academic curiosity. They open up capabilities that language models structurally can't provide. An agent that can simulate "if I take this action, what state results?" can plan over many steps instead of guessing one at a time. A model of normal dynamics flags the abnormal without needing a labeled example of every failure mode, which covers most of anomaly detection. Forecasting how a process, a fleet, or an environment evolves is a world-modeling problem, not a text problem. And agents that carry a model of their environment need far less interaction with it to behave well, which is what sample efficiency means in practice.

We built a tiny one, and it kept us honest

Claims like "predict in representation, not in pixels" deserve numbers, so we ran a small experiment: a damped pendulum observed through 64 noisy sensors, and two predictors trained on identical data. The raw path is a ridge regression mapping all 64 sensors straight to all 64 sensors, 8,192 parameters. The latent path encodes each observation into 16 numbers with PCA (self-supervised, no labels), steps forward with a 256-parameter dynamics model, and decodes.

The core of the latent path is small enough to quote in full:

mu = X.mean(axis=0)                      # X: observation pairs, 128 dims
_, _, Vt = np.linalg.svd(X - mu, full_matrices=False)
E = Vt[:16].T                            # encoder: 128 -> 16, no labels used
Z_now  = (X - mu) @ E
Z_next = (X_next - mu) @ E
W = ridge(Z_now, Z_next, lam=1e-3)       # the entire world model: 16 x 16

Both models were rolled out autoregressively on 30 held-out trajectories and scored on decoded angle error, at two sensor-noise levels:

The two lines are the point: orange is the 256-parameter latent model, gray is the 8,192-parameter raw model, and they land within a few percent of each other everywhere, with the latent model slightly ahead at mid horizons once noise gets heavy. Thirty-two times fewer dynamics parameters, no labels, same accuracy. That is "spend capacity on structure" as a measurement rather than a slogan.

Then we tested the anomaly claim, and it failed. We changed the pendulum's physics mid-trajectory (gravity jumps from 9.8 to 25, a stand-in for a real fault) and watched one-step prediction error for a spike. Nothing. The fault's per-step signature is about 0.03 radians; our model's ordinary per-step bias is ten times that, so the signal drowned. A ten-step detector failed for the mirror reason: the linear model's own rollouts drift half a radian on normal data, because a pendulum's period depends on amplitude and no linear map can represent that. Upgrading to cubic features cut the bias by a third. Still not enough.

We're publishing the failure because it teaches the sharpest lesson in the experiment: anomaly detection by world model is bounded by simulator quality. The signal was there. Our model was too biased to see it. The full code, data generation, and verbatim results are in our repo's experiments directory.

The honest state of the art

We want to be precise about where this stands. World models are an active research frontier, not a solved technique you drop into production. Training stable predictive representations, avoiding collapse, and transferring them to real operational domains are open problems the whole field is working on. The results that exist are early and domain-specific.

Our toy experiment above is that frontier in miniature. The representation-efficiency claim held at first try. The anomaly claim needed a better simulator than a linear model can be. Closing that gap, at production scale and in domains messier than a pendulum, is the work.

That is exactly why the work is worth doing. The teams that learn to model dynamics, and not merely describe them, will build AI that plans, anticipates, and operates. Everyone else will build AI that only answers.

At ArthaVortex, world models sit at the center of our research, alongside the multi-model systems and agents that put learned representations to use. It's the part of the field we're most convinced is underexplored relative to its importance.

Related: Beyond the Language Model on the broader shift, and Beyond Flat Embeddings on the representations these models learn in.