Text this: Extremal events dictate population growth rate inference