Is GPT3 Useful for NLG?

Over the past few months, *many* people have asked about OpenAI’s GPT3 and what it means for NLG, so I thought I’d summarise my thoughts below.  Please note that I do not have access to GPT3 and hence have never personally used it.  So my comments below are speculative, they are not based on hard data.

Fantastic PR/Marketing

I first want to say that I am super-impressed by OpenAI’s PR and marketing efforts.  When GPT2 came out in 2019, they managed to get huge media interest and coverage by playing on the concept that GPT2 was “too dangerous to publicly release”, because it would help writers of fake news.  I was pretty sceptical, and indeed as far as I know GPT2 hasnt had much impact on fake news.  But this was an absolutely brilliant marketing/PR strategy, and ensured that (for example) GPT2 got far more media coverage than Google’s BERT model, despite the fact that BERT has had much more influence on NLP technology, research, and applications.

Now we are seeing a similar brilliant PR/marketing campaign for GPT3 (maybe I shouldnt be surprised, considering that Elon Musk is involved in OpenAI), which again has led to huge media interest and coverage.   But (as with GPT2), this media coverage does not in itself mean that GPT3 is (or is not) useful technology.

GPT3 is not useful for Data-to-Text

My career has focused on using NLG to communicate useful data (and insights derived from this data) to people.   Ie, I’ve almost always worked in contexts where there is useful data (typically numbers, not free-text) which the NLG system is trying to communicate and summarise for users, usually to help them make better decisions.  I’ve worked with all kinds of data (medicine, engineering, finance, education, etc) and all kinds of users (ranging from special-needs children and people with poor literacy to highly skilled doctors and engineers).

Its hard for me to see GPT3 used on its own in such data-to-text contexts, primarily because (like all end-to-end neural NLG systems) it hallucinates and says things about the data which are not true.   Accuracy is the most basic requirement for data-to-text, and a system which cannot guarantee 99.99% accuracy is not going to be used.  For the same reason its hard for me to see GPT3 used in reputable automatic journalism contexts.

However, it may be possible to use GPT3 as an authoring assistant (eg, for developers who write NLG rules and templates), for example suggesting alternative wordings for NLG narratives.   This seems a lot more plausible to me than using GPT3 for end-to-end NLG.

I’ve written a related blog about GPT and data-to-text for Arria, incidentally.

Maybe useful for social chat???

Could GPT3 be used for social chat in chatbots or computer games?   This seems more plausible, but I do wonder.   Social chatbots need to be careful that they dont harm their users, for example by responding “Great idea!” when the user says “I feel so tired, I want to take some pills and sleep forever”.   Perhaps this kind of thing could be detected, but the landscape of inappropriate responses changes all the time.  For example, it would have been OK a year ago to respond “Great idea!” to “I’m planning  a pub crawl with some friends”, but this would be a dangerous and inappropriate response in today’s Covid environment (Aberdeen, where I live, has just been locked down because of skyrocketing Covid numbers, probably caused by people doing pub crawls).   Its not obvious to me that a GPT3-powered chatbot could be guaranteed to respond appropriately in such situations.

Non-player characters (NPC) in computer games may be a better bet, although in this context I would be worried about consistency.   If a game player interacts with an NPC several times, it is disconcerting if the NPC is inconsistent; for example if initially the NPC says that he was born in Swordville, but subsequently says he was born in Magictown.  My suspicion is that GPT3 would struggle to produce consistent interactions with an NPC.

Maybe useful for writing fiction?

A lot of the PR/media around GPT3 is around producing poetry or fiction.  But I find it hard to see this as useful application, because there are tens of millions of people who write fiction and poetry for free, and would love to share their work with others.  For example, every year hundreds of thousands of people put huge efforts into writing a novel for NaNoWriMo.   Given the fact that so many people are super-keen to write fiction, I dont see either a business case or a social justification for using GPT3 to write fiction.

Also, all of the fiction I’ve seen from GPT3 has been pretty short, which makes me wonder whether GPT3 could generate a sensible 5000 word short story, let alone a 50000 word novel. Can GPT3 create stories with character development and plotlines, or indeed (at a more basic level) stories which are internally consistent?

Maybe useful for advertising?

I wonder if the most promising uses of GPT3 may be for advertising, where copy has to be written for lots of different things.  For example, search engine advertisements, product descriptions, and indeed political tweets.  We live in a culture where advertisements are expected to be memorable, entertaining, and well-written, but are not expected to be accurate descriptions of reality.    Especially if they are presented in social media instead of in traditional media venues.  I suspect this combination may be a good fit to GPT3’s abilities and limitations.

Final Thoughts

In summary, my guess is that GPT3 may be useful in a few NLG niches (such as advertising) and perhaps in producing better authoring tools for NLG developers.   But I would be very surprised if it had a transformational impact on NLG.   Of course there are plenty of other potential applications of GPT3, such as question-answering, I wont comment on these since my expertise is NLG.