Principled Penalty-based Methods for Bilevel Reinforcement Learning and RLHF (Journal of Machine Learning Research, 2026)