All Stories

Even a Bag of Words Has a World Model

Skeptics call linear probes that recover geography from an LLM 'mere statistics.' But compression is the discovery of structure — even a plain bag-of-words model of text builds a world...

Thoughts on inscriptions

“Inscriptions” became a popular way to create NFTs on Bitcoin earlier in 2023 and have picked up significantly in the past month. However, they are, in many ways, a step...

Parallel scans

This post is inspired by the recent paper on Mamba. Mamba introduces a simplified, linear RNN and shows that it can be computed in \(\mathcal{O}(\log n)\) time using a parallel...