Similar Items: Extremal events dictate population growth rate inference