GPT v BERT in Dev Survey

• By John K. Waters
• 12/05/2023

The results of a new survey suggests that one of the two major architectures for Natural Language Processing (NLP) is edging the other in popularity among AI and machine learning developers, though both command a significant number of fans.

In Evans Data’s semi-annual "Artificial Intelligence and Machine Learning Development" survey, conducted last month, developers were asked whether an autoregressive model, such as GPT, or a bidirectional model, such as BERT, is best suited for their NLP projects. 43% chose the autoregressive model, which predicts the next word in a sequence, while 40% preferred the bidirectional approach, which assesses words in both directions for a deeper language understanding.

GPT (Generative Pre-trained Transformer) is, of course, the underlying language model on which OpenAI's enormously popular ChatGPT is based. GPT predicts the next word in a sequence given all the previous words within some text.

Further from the current spotlight, BERT (Bidirectional Encoder Representations from Transformers) reads the entirety of the language bidirectionally. This allows the model to learn the context of a word based on all of its surroundings (both left and right of the word). Google introduced BERT in 2018.

In the Evans survey, conducted twice yearly, aims to provide insights into current trends in AI and machine learning development.

In the survey, developers were more likely to cite using GPT4 (the latest iteration) than any other specific language model, although more versions of BERT were cited. Search engines, customer service applications, grammar correction, and legal research were highlighted as prime NLP applications.

The Even AI/ML development durvey is conducted in the spring and fall and delivers more 180 fresh data points about what developers are doing on up to the minute subjects such as neural networks and transformers, generative AI, deep learning methods, computer vision and image recognition, NLPs, machine learning frameworks and much more detailed comprehensive information.