Turing-NLG – Natural Language Generative model by Microsoft

Microsoft released an NLG which is massive deep learning language model such as BERT and GPT-2, with billions of parameters learned from essentially all the text published on the internet. This improved the state-of the art natural language processing (NLP) task, including question answering, conversational agents, and document understanding among others.

Better natural language generation can be transformational for a variety of applications, such as assisting authors with composing their content, saving one time by summarizing a long piece of text, or improving customer experience with digital assistants.

Turing Natural Language Generation (T-NLG), is the largest model ever published at 17 billion parameters, which outperforms variety of language modeling benchmarks and also excels when applied to numerous practical tasks, including summarization and question answering. This work would not be possible without breakthroughs produced by the DeepSpeed library (compatible with PyTorch) and ZeRO optimizer.

This figure was adapted from a similar image published in DistilBERT.

T-NLG is a Transformer-based generative language model, which means it can generate words to complete open-ended textual tasks. In addition to completing an unfinished sentence, it can generate direct answers to questions and summaries of input documents. T-NLG is important for NLP tasks since the goal is to respond as directly, accurately, and fluently as humans can in any situation. we can naturally summarize or answer questions about a personal document or email thread.

The bigger the model and the more diverse and comprehensive the pretraining data, the better it performs at generalizing to multiple downstream tasks even with fewer training examples. Therefore, it is more efficient to train a large centralized multi-task model and share its capabilities across numerous tasks rather than train a new model for every task individually.

T-NLG future applications

T-NLG has advanced the state of the art in natural language generation, providing new opportunities for Microsoft and their customers. Beyond saving users time by summarizing documents and emails, soon you will see Microsoft Office suite with T-NLG by offering writing assistance to authors and answering questions that readers may ask about a document. Furthermore, it paves the way for more fluent chatbots and digital assistants, as natural language generation can help businesses with customer relationship management and sales by conversing with customers.

I am currently learning/practicing on ZeRO and DeepSpeed. Will share the learning in my next post.