Looped Transformers Shown to Emulate General-Purpose Computers
The paper presents a framework for programming transformer networks with specific hand-crafted weights, then running them in a loop, so the input sequence acts like a punchcard encoding instructions and memory. The authors show a constant number of encoder layers can implement basic computational primitives (edit operations, nonlinear functions, function calls, program counters, conditional branches), and that a 13-layer looped transformer can emulate a small instruction-set computer capable of running a calculator, a linear algebra library, and even in-context learning via backpropagation. This suggests that shallow transformers, when looped, are theoretically capable of executing arbitrary general-purpose programs. Twitter discussion was brief, mainly flagging the paper as a notable result on the computational universality of looped transformers, with no substantive criticism raised in the visible commentary.
Discussion: 1 tweets from 1 authors · @jasondeanlee