Similar Items: Misaligned by Reward: Socially Undesirable Preferences in LLMs