Thanks for your question George. We used the masked modeling objective. We didn’t use the Next Sentence Prediction (NSP) task as a lot of recent work (e.g. RoBERTa) has shown that it is not that valuable.