AI Castaway

Everything You Need To Know About Private LLMs

The consumerization of AI has a lot of organizations yelling, "Privacy, please!" An LLM grounded on internal data might just be the answer to those AI privacy and security concerns.

• By Ginger Grant
• 07/31/2024

For many organizations, having a large language model (LLM) answer questions about your data using your data is an appealing concept. The catch is, current LLMs are perceived as full of errors.

A recent study of manufacturers shows companies are not sure that they want to incorporate LLMs in their business. One of the largest issues organizations have with using LLMs is their accuracy -- or lack thereof. In short, organizations need LLMs to be more accurate to be useful.

One way that some companies are improving accuracy is by grounding LLMs with their internal data. This is called a private LLM -- an LLM that leverages an organization's own data to answer internal users' questions. The grounding data can include proprietary strategies and methods specific to your organization, company data, industry information and product details. Training an LLM on your data grounds the LLM on information that's more relevant to your users than a general-knowledge database, resulting in more useful responses. For example, a private LLM-powered chatbot can provide answers about current customers or the status of ongoing projects.

One big caveat to this method is security. An LLM that uses a company's internal data as a training source needs to prioritize privacy to ensure that, for example, the company's proprietary product and design strategies remain secure. Executed correctly, however, private LLMs can provide more accurate, relevant and specific information.

Two Techniques for Creating Your Own Private LLM
There are two different methods for creating a private LLM:

• Training with your own data.
• Incorporating your own data with retrieval augmented generation (RAG).

The first method is more expensive as it takes considerably more effort. Training LLMs requires a lot of data processing and compute power -- and, of course, people who have the necessary skills to develop a useful model.

The RAG method is a little less complicated as it entails using an existing model, like Anthropic's Claude or Meta's Llama, and supplementing it with your data. The model will still use data it was originally trained on, but it will also be able to answer questions about your product lines, for example.

The big challenge to the effectiveness of both of these methods is data. Your organization needs to ensure it has a large index quantity of internal data available to train the model. Useful data needs to be gathered, formatted and trained to be helpful. You will also need to establish an ongoing process to update the LLM whenever there's a change to your data. Devoting resources to stay current can be challenging given the rapid pace of development in the AI space.

On top of developing and updating your model, having a private LLM generally means that you have to create a private chatbot to interact with it, as well.

Where Private LLMs Are Being Created
No matter which method you use, creating an LLM is going to require a lot of compute power.

Some organizations consider their data to be highly proprietary; the data exists only on internal servers and is developed internally on their own servers, as well. This strategy requires a significant investment in compute hardware.

Private LLMs can also be created on cloud servers. Depending on the provider, the data can be stored securely in the cloud and the provider manages the hardware.

There are also a number of companies that offer to create and store your private LLM using specialized tools to assist in the creation.

A Small Selection of Private LLM Providers
As you might expect, several companies have sprung up to specialize in the creation of private LLMs. Here are just a few:

• Lucidworks started as an enterprise search company over a decade ago and now provides private LLM development services.
• Glean gathers data from the entire organization and uses it to develop a chatbot assistant to answer questions about that data.
• Blankfactor develops private LLMs for the financial industry.
• Tonkean has a low-code method for developing private LLMs.
• Tursio specializes in gathering information from databases and creating LLMs using that information.

All of these companies focus on providing organizations with insights about their data outside of traditional tools such as reporting visualizations. Some speculate that interacting with data using private LLM-based methods may supplant reports as the primary method organizations use to understand key internal metrics.

Private LLMs in the Wild
Apple is currently working on a chatbot codenamed "Ajax" that is meant to be a private LLM. Anyone who has an iPhone will be able to use it once it is released. The LLM is trained by the phone user over time. The training data will remain on the phone; Apple promises that it will not use it to train other models, only to fine-tune data. The company is adding it to the OS as an on-device LLM to preserve user privacy.

There are a number of other high-profile companies currently using private LLMs, including JPMorgan Chase, Morgan Stanley, Goldman Sachs and Deloitte. All of these companies have implemented private LLMs to assist their employees.

There are other companies working on their own private LLMs, with some of them trying and ultimately failing. Unfortunately, I could not get any of those companies to admit that for Internet publication.

Summary
So, should your company invest in a private LLM?

Having a chatbot that you can ask about things that are specific to your organization -- like product or customer status -- using regular language is a very appealing proposition. However, given data from various surveys and anecdotal stories about failed private LLM implementations, it is a risk.

Take, for example, this recent survey that suggests companies are doubtful of the theoretical value that an AI project can provide. The costs of development and implementation of private LLMs are high, and the success rates do not show that the benefits are there yet.

But this, like everything in AI, is bound to change.