Similar Items: Principled Penalty-based Methods for Bilevel Reinforcement Learning and RLHF