Monday November 9, 2020 0 comments
By Thomas Frey
The DaVinci Institute
Elon Musk has given us nightmarish warnings about how AI could become ‘an immortal dictator from which we would never escape.’
However, GPT-3 has been created by OpenAI, a research business co-founded by Elon Musk and has been described as the most important and useful advance in AI for years.
What is GPT-3?
GPT-3 stands for Generative Pre-trained Transformer 3 -- it’s the third version of the tool to be released.
In short, this means that it generates text using algorithms that are pre-trained -- they’ve already been fed all of the data they need to carry out their task. Specifically, they’ve been fed around 570gb of text information gathered by crawling the internet (a publicly available dataset known as CommonCrawl) along with other texts selected by OpenAI, including the text of Wikipedia.
If you ask it a question, you would expect the most useful response would be an answer. If you ask it to carry out a task such as creating a summary or writing a poem, you will get a summary or a poem.
More technically, it has also been described as the largest artificial neural network ever created -- I will cover that farther down.
What can GPT-3 do?
GPT-3 can create anything that has a language structure -- which means it can answer questions, write essays, summarize long texts, translate languages, take memos, and even create computer code.
In fact, in one demo available online, it is shown creating an app that looks and functions similarly to the Instagram application, using a plugin for the software tool Figma, which is widely used for app design.
This is, of course, pretty revolutionary, and if it proves to be usable and useful in the long-term, it could have huge implications for the way software and apps are developed in the future.
As the code itself isn’t available to the public yet (more on that later), access is only available to selected developers through an API maintained by OpenAI. Since the API was made available in June this year, examples have emerged of poetry, prose, news reports, and creative fiction.
How does GPT-3 work?
In terms of where it fits within the general categories of AI applications, GPT-3 is a language prediction model. This means that it is an algorithmic structure designed to take one piece of language (an input) and transform it into what it predicts is the most useful following piece of language for the user.
It can do this thanks to the training analysis it has carried out on the vast body of text used to “pre-train” it. Unlike other algorithms that, in their raw state, have not been trained, OpenAI has already expended the huge amount of compute resources necessary for GPT-3 to understand how languages work and are structured. The compute time necessary to achieve this is said to have cost OpenAI $4.6 million.
To learn how to build language constructs, such as sentences, it employs semantic analytics – studying not just the words and their meanings, but also gathering an understanding of how the usage of words differs depending on other words also used in the text.
It’s also a form of machine learning termed unsupervised learning because the training data does not include any information on what is a “right” or “wrong” response, as is the case with supervised learning. All of the information it needs to calculate the probability that it’s output will be what the user needs is gathered from the training texts themselves.
This is done by studying the usage of words and sentences, then taking them apart and attempting to rebuild them itself.
Will AI replace humans? All the scary articles fail to answer the basic question of “Why?”