Leveraged Learning: entropy destroyed per bit received

Abstract

A learner holds a prior belief over boolean maps that answer a finite set of $Q$ questions, and receives answers one by one. Each answer costs surprisal and destroys uncertainty, not only about the question asked but every question still unasked. We call the ratio the leverage: table entropy destroyed per bit of surprisal received. It is unit for a uniform prior, but with an intelligent prior can be higher (not lower). Averaged over the truth prior and over the question order, the leverage is found exactly, and is generated by one sequence: the mean entropy $G_\ell$ of the answers to $\ell$ questions. Sending the number of input bits to infinity at fixed asked fraction $t = \ell/Q$, the increments of that sequence become a profile $\gamma(t)$, and initial question entropy $\eta_0$. The leverage closes to a thermodynamic limit. $L(t) = [\eta_0 - (1-t)\gamma(t)]/\int_0^t \gamma$. Exchangeable priors, by de Finetti, all give a flat $\gamma(t)$ and hence a hyperbolic $L(t)$, their deduction confined to a boundary layer at $t = 0$. We construct a simplicity prior that escapes this, grading Boolean maps by the degree of their polynomial over $\mathbb{F}_2$ and budgeting weight across degree shells by a CDF $F$. Reed-Muller capacity then gives $\gamma(t) = 1 - F(t)$ exactly, so any nonincreasing profile, and any leverage curve it generates, is realizable at macroscopic times.