my work, AI-generated overview

Somewhere around 2015 I admitted to myself that my IMDB ratings were garbage: the numbers meant nothing. Asked to give a movie 7 or 8 stars, I produce noise: it depends on my mood, on the last thing I rated, on whether I’m anchoring against masterpieces or against the mediocre thing I watched on a plane. A 7 from 2013-me and a 7 from 2015-me are not the same 7. What I wanted was for my ratings to be calibrated: an 8 should be a statement with content, comparable across years and genres.

But here’s the thing: while I can’t rate a movie in isolation to save my life, ask me which of two movies is better and I usually answer quickly, and consistently. This is an old observation — psychophysicists have known for a century that absolute judgments drift and paired comparisons are stable. So the plan wrote itself: stop rating. Compare, and let the machine infer the numbers.

The machine works like this. Every item gets a hidden score, and a comparison is a noisy coin flip whose bias depends on the score difference — the same family of models behind Elo in chess. The twist is to be properly Bayesian about it: instead of a single number per movie, keep a full posterior distribution over every score, so the system knows not only the ranking but how sure it is of each part of it. That uncertainty then gets put to work: the system picks the next question to ask — which two movies to show me — to learn as much as possible from each answer. And at the very end, stars fall out for free: map the posterior scores through their quantiles and an 8 finally means “better than this fraction of everything you’ve compared,” calibrated by construction.

In 2022 I finally sat down to build it. The reason it turned into a project rather than an afternoon is a single integral. The Bayesian fit needs the expected value of a certain smooth loss under a Gaussian, and that integral has no closed form — the standard move is to approximate it numerically, everywhere, millions of times. But it turns out the logistic curve is almost an error-function curve, and error functions have a lovely property: the approximation commutes with the expectation. Approximate first or average first, you get the same expression. Add a small correction — a handful of Gaussian bumps, fitted once — and the approximation error drops to three parts in ten million while every quantity you need for optimization stays in closed form. The whole posterior fits with textbook Newton iterations, no sampling, no quadrature. As I wrote in the paper, the closed form has no business working out as cleanly as it does. But it does, and chasing that kind of unreasonable tidiness is most of why the project is fun.

The empirical side had its own surprise. Of the strategies for picking the next comparison, the greedy one — directly minimize the expected number of ranking mistakes, the very metric being scored — turns out to be the worst, losing even to picking pairs at random. It keeps probing the current photo-finishes instead of spreading its questions around. The strategies that target uncertainty itself, like expected information gain, win comfortably. Optimizing your objective one step at a time is not the same as optimizing it.

The project then sat mostly dormant for a few years, the way side projects do. This year I picked it back up — coding agents have lowered the activation energy of “finally do it right” to roughly zero — and finished the job: a proper write-up of the math, a test suite, and support for ties, because not every comparison has a winner. Ties brought one more pleasant surprise: a 1970 model by Davidson slots into the same closed-form machinery exactly, down to the integrals, with the tie-proneness of the domain (rare for movies, common for chess, unknown a priori) learned from the data alongside everything else.

The code (a reference Python implementation and a fast C++ one) and the paper are at github.com/murbard/ranker. As for the original itch: a decade later I still haven’t re-rated my movies. But the difference between “my ratings are garbage” and “my ratings are garbage, and I have a machine that fixes this” matters to me, and the second one is now true.