Text this: It Just Takes Two: Scaling Amortized Inference to Large Sets