my work, AI-generated overview

In October 2023, Gurnee and Tegmark published Language Models Represent Space and Time: linear probes on Llama activations recover, among other things, the latitude and longitude of places the model has read about. The paper was received as ammunition in the “do LLMs have world models?” debate, and the skeptics’ rebuttal was predictable: recovering coordinates with a linear probe is mere statistics, correlations between surface features, not a model of the world.

I always found that rebuttal strange, because it gets the burden of proof backwards. Of course LLMs have world models. Modeling the world is not an exotic capability that emerges at some parameter count; it is what learning from data about the world does by default. Compression is the discovery of structure. The interesting questions are about how rich and how faithful the model is, not whether one exists.

Back then I started a small experiment to make the point by reductio, and, in the age of coding agents, finally finished it. Strip everything away — no transformer, no embeddings, no attention, no pretraining. Take the English Wikipedia article of every country and territory, compute tf-idf, and fit a linear ridge regression from word frequencies to the country’s location. If a linear readout of geography counts as evidence of a world model, then a bag of words has one too.

To keep it honest: 238 of the 249 articles literally contain the country’s coordinates (Wikipedia footers), so coordinate strings are stripped and tokens containing digits are excluded from the vocabulary. And rather than regressing latitude and longitude directly — which punishes predictions near the antimeridian with an absurd squared loss — the model predicts a point in 3D that gets projected back onto the sphere. Everything is evaluated with 10-fold cross-validation, vectorizer refit on the training folds.

The result: a median error of about 1,050 km across 248 countries and territories.

The mistakes are the best part, because they have geographic semantics rather than the flavor of noise. Réunion is pulled toward France and Timor-Leste toward Portugal; Kiribati, which straddles the antimeridian, lands on the wrong side of the Pacific; Canada and Chile get dragged toward their population centers. The regression is wrong the way you would be wrong if everything you knew about the world came from reading about it.

Two details surprised me. In 2023, my quick attempts had gradient boosting comfortably ahead of linear regression; once the target lives on the sphere, the linear model wins (1,058 km vs 1,344 km median). And cross-validation selects the smallest ridge penalty on the grid in every single fold: with ~30,000 features and ~220 training examples, the best regime is near-interpolation. The geographic signal isn’t concentrated in a handful of gazetteer tokens that regularization would preserve; it is smeared diffusely across the entire vocabulary.

If a linear map from word counts recovers the shape of the globe, then “the probe is just statistics” fails as a rebuttal — not because the probe is more than statistics, but because there is no distinction to draw. Statistics about the world, compressed, is a model of the world. The bag of words has a crude one. LLMs have vastly better ones. The claim that they have none was always outlandish.

Code, data, and the experiment log: github.com/murbard/tfidflatlong.