Text this: Why Self-Supervised Encoders Want to Be Normal