In-Depth

AI Coding Assistants Shake Up Software Development, But May Have Unintended Consequences

• By Pure AI Editors
• 02/03/2023

OpenAI Codex and GitHub Copilot are closely related AI systems that can automatically write computer code snippets and even entire computer programs. Codex/Copilot have the potential to revolutionize software development. But experts caution there will likely be unintended consequences -- both positive and negative.

As its name suggests, Codex is a system developed and maintained by OpenAI. Copilot is a system that uses Codex technology. Copilot is developed and maintained by GitHub. GitHub is owned by Microsoft.

Briefly, a Transformer is a complex low-level code module. GPT-3 is a large language model that uses Transformer architecture, is trained on Wikipedia, and that understands English. ChatGPT is a conversation chatbot application built on top of GPT-3. Codex is a system with ChatGPT plus additional training on computer code so Codex understands English and computer languages. Copilot is a wrapper over Codex that integrates directly into a software development programming environment.

Codex
Codex is an extension of ChatGPT. ChatGPT is a natural language chatbot that can produce astonishing results. ChatGPT is based on the GPT-3 ("Generative Pre-Trained Transformer") large language model. GPT-3 was trained on an enormous corpus of text -- Wikipedia, books, blogs and so on. ChatGPT understands the English language (in a way that's not fully understood) and can respond to requests such as, "Write two paragraphs about the history of chess."

Codex is an extension of ChatGPT and GPT-3 where additional training was applied, using billions of lines of computer code. So in addition to natural language, Codex also understands computer code in several common programming languages such as Python, JavaScript, C#, C++, HTML, CSS and SQL.

Software developers can use Codex by typing a query into a text box on a web page. A Codex query might be, "Show me an array binary search function using Python" or "SQL code to create a database of books." The Codex result is code that a software developer can copy-paste into an integrated development environment (IDE) such as Microsoft Visual Studio or Pycharm. See Figure 1.

Copilot
GitHub Copilot is essentially a wrapper over OpenAI Codex. Copilot is integrated directly into a development environment, most often the heavyweight Microsoft Visual Studio or the leaner Visual Studio Code. Using Copilot can be a bit more efficient than Codex because the Copilot code results go directly into a program instead of having to copy-paste. For example, in Visual Studio, a developer can type a comment such as, "// code for a list of color items" and Copilot will respond with:

public class ColorItem
{
  public string Color { get; set; }
  public string Name { get; set; }
}

See Figure 2 for an example of Copilot being used for a C# language program in Visual Studio. Both Codex and Copilot have free trial periods, but are paid services. For example, Copilot currently costs $10 per month.

The Impact of Codex and Copilot
In a typical software development scenario without Codex/Copilot, a developer has an IDE open on the desktop, and also a web browser. There is continuous back and forth activity where the developer issues web search queries for code examples and then after reviewing half a dozen or so results from blogs, the Stack Overflow web site and technical documentation, the developer copy-pastes code from the web page into the IDE.

Before the internet, software developers relied on books. The software development process was much, much slower because it often involved a trip to a library or bookstore. The internet dramatically increased the speed of software development, roughly by a factor of 10. Codex/Copilot have the potential to increase the current speed of software development again, perhaps by another factor of 10.

In their current stage of evolution, Codex and Copilot appear to be highly effective for queries where the correct response is relatively short (roughly 20 lines of code or less) and well-known. Put another way, Codex and Copilot are highly efficient resources for code examples and code syntax.

Unintended Consequences
If AI systems like Codex and Copilot are widely adopted, there will likely be unexpected and unintended consequences. One possible negative consequence is an increased frequency of serious programming bugs. Codex and Copilot results are not guaranteed to be correct. Even though Codex/Copilot code results are just suggestions, software developers are just ordinary humans (well, mostly) and there will be a strong tendency to accept a code suggestion without thoroughly analyzing it.

Another possible negative consequence of using AI-assisted programming is that the process may impact software developer skill set acquisition. In a current scenario where a developer parses through several blogs and documentation examples, there is active learning going on. Each query-search-code instance increases a developer's personal skill set. But with an automatic AI-based system, no such learning-by-doing is happening. The ability of a developer to effectively use Codex or Copilot depends on being able to ask the right questions.

The Pure AI editors spoke with Dr. James McCaffrey from Microsoft Research. McCaffrey commented, "Like most technologies, AI assisted programming will have pros and cons. Like it or not, systems like Codex and Copilot are probably here to stay."

He added, "These AI assistant systems are all based on existing knowledge. In their current states, they can simulate creativity in very surprising ways by combining ideas but they really can't generate completely new algorithms."

McCaffrey further noted, "One of my work colleagues speculated about a future with many specialized AI assistant systems, such as a physics assistant, a biology assistant and so on. Imagine if all these AI assistants could automatically communicate with each other using natural language via ChatGPT. It's fun to imagine what that scenario could lead to."