$f$-Divergence Regularized RLHF: Two Tales of Sampling and Unified Analyses (ArXiv cs.IT Recent Papers, 2026)