Global Data Science Forum

GPT-3 Release: Thoughts & Resources

By Michael Mansour posted Wed August 05, 2020 05:31 PM


OpenAI’s GPT-3 release in private-beta this July brings the largest language model to the NLP community to date.  Now that the community has played around with it, we’re starting to see the model’s amazing outputs.

GPT-3 boasts 175B parameters; recall that GPT-2 had 1.6B parameters.  Aside from this massive model scale increase, GPT-3 doesn’t bring many other groundbreaking innovations.  With such a highly parameterized model, they claim that fine-tuning for different NLP task contexts outside of language generation is no longer needed.  In fact, it performs very well in zero/few-shot tasks it was not even trained on. Furthermore, all those parameters appear to have learned facts about the world and even programming languages in the training corpus.  Off the shelf then, it can do Q&A and even generate functional javascript from a natural language prompt.  This all comes at a cost: Someone did the math and estimated that GPT-3 would take at least $4.6 to train, and the weights would require ~700GB of RAM. These resources are not generally available to most companies, and would further increase the difficulty to fine-tune the model.

To catch up on this release with specific topics relevant to you, see the links below:

My Thoughts 

It’s not clear when GPT-3 will actually become available to the public.  Given that there was a large debacle about GPT-2 being too dangerous to release, that debate is likely to intensify with this latest model.  I wonder how OpenAI is determining who gets access to the private beta release, and how they’ll fairly delegate that access in the future. Should we be trusting the creators of potentially dual-use technology to be in charge of that, or should it be relegated to a third-party organization like the Apache/Linux Foundation, or some group thought-leading AI-Safety/Ethics?  Does at some point the release of powerful language models like this become a national security issue, and the government can impose export controls?  These are important questions we should be asking ourselves.