Principled Penalty-based Methods for Bilevel Reinforcement Learning and RLHF (JMLR, 2026)