Language Models have dominated NLP literature since Vaswani et al seminal paper on Attention Mechanisms, now known as Transformers. The talk will compare and contrast the popular fine-tuning frameworks GPT-2 [left to right transformer] and BERT [bidirectional transformer], review the underyling assumptions each model makes, their use-cases, and their impact beyond NLP.