Vision Transformers Repurpose Background Patches as Scratch Memory — Fix: Add 'Register' Tokens
The paper identifies that supervised and self-supervised Vision Transformers develop high-norm 'artifact' tokens, typically in low-information background patches of images, which the network appears to repurpose as internal scratchpad storage during inference rather than for their original spatial role. The authors propose adding extra, dedicated 'register' tokens to the input sequence to absorb this internal computation, which eliminates the artifacts entirely, smooths attention and feature maps, improves object discovery with larger models, and sets a new state of the art for self-supervised dense prediction tasks. Twitter commentary highlighted this as conceptually similar to the observation that LLMs sometimes use filler tokens (like periods) as computational scratch space to improve performance, suggesting a shared pattern where transformer models exploit spare tokens as working memory across modalities.
Discussion: 1 tweets from 1 authors · @yu4u