Text this: $f$-Divergence Regularized RLHF: Two Tales of Sampling and Unified Analyses