THE MEDIUM UNLOCK TIME MOVIE

This movie is …”, GPT3 is much more likely to output the word “bad” compared to “good”. When continuing the sentence “While the cast was okay, the script of the movie was horrible. It turns out that being good at generating coherent natural language has many benefits. Given an input sentence, GPT3 continues it in a natural way. As a result, GPT3 is currently the best-in-class model for generating human-like language. Because this task requires no additional annotations (we call it self-supervised), researchers were able to gather large amounts of data (40GB of text) and train a ridiculously large deep learning model (175 billion parameters). GPT3, one of these models, was trained by predicting the continuation of a given sentence as accurately as possible (this task is called causal language modeling). Recently, large pre-trained generative models are all the rage. Interestingly, a new paradigm seems to be emerging in this field which could have a major impact on how we will use deep learning in the future. The field of Natural Language Processing (NLP) has undergone these evolutions as well. Eventually, data scientists created techniques for applying large pre-trained models to many downstream tasks via transfer learning, which allowed these models to attain strong performance on many tasks with only a seemingly small amount of data. With the advent of deep learning, models are now able to extract their own quality features given heaps of data.

Prompt-engineering: the new deep learning paradigm?īack in the old days of machine learning, engineers had to spent countless hours creating informative and discriminative features to improve the quality of their models. If you don’t feel like reading, you can watch this recording of the internal meeting where I presented my solution to the company. Finally, I will conclude with some remarks on this new and exciting trend. The solution works with extremely few datapoints and can easily be adapted towards new situations without retraining. I will present a case study where I used this technique during my summer internship at Waylay to create an application that makes industry-level automatization accessible to everyone using voice and text inputs (think of something like google assistant, but for IoT and on steroids!). This new paradigm allows us to rapidly prototype complex NLP applications with little to no effort and based on very small amounts of data. The advent of large pre-trained language models (such as GPT3 ) are giving rise to the new paradigm of ‘prompt engineering’ in the field of NLP. Photo by Alfons Morales on Unsplash Introduction

The medium unlock time