Leveraging LLMs for Competitive Advantage: What Every CTO Needs to Know

With AI at the forefront of business innovation, choosing the right model has become a critical and complex decision. The options are vast and multiplying by the day. Giants like OpenAI, Google, and Meta have released groundbreaking models—trained on datasets approximating the entirety of human knowledge and backed by billions in R&D. You've probably heard of them, and chances are, you've even used them. As businesses look to implement AI to impact the bottom line and as entrepreneurs rush to create the next innovative product, the question arises: what models, or specifically LLMs, should you be using?

What Is an LLM?

So, what exactly is an LLM, or Large Language Model? In simple terms, an LLM is a type of AI trained on vast amounts of text data to understand and generate human language. These models use deep learning techniques to process and predict text, enabling them to perform tasks like text generation, translation, summarization, and even answering questions. Some well-known examples of LLMs include OpenAI's GPT-4, Google's Gemini, and Meta's LLaMA models.

How Do They Work?

Now, this isn’t meant to be a technical deep dive, but let’s touch on the basics of how these models work. AI is a big, broad field encompassing a lot of stuff—a rabbit hole that starts with machine learning. At its core, machine learning is a type of statistical algorithm that can learn without explicit instructions, enabling it to identify patterns in data and make predictions or decisions based on them. The larger the dataset and the more complex the predictions, the more sophisticated the model needs to be.

At a high level, these models, known as neural networks, are computational models inspired by the human brain, consisting of layers of interconnected nodes. In 2017, some smart folks at Google introduced a game-changing concept called transformers in a paper that’s since become famous. Transformers revolutionized the way we process data by allowing us to process all elements in a sequence simultaneously, rather than sequentially, which was the norm before. They also introduced the attention mechanism, which lets models focus on specific parts of the input sequence that are most relevant to making predictions. This breakthrough led to a massive leap in Natural Language Processing (NLP) performance, with models like GPT, LLaMA, and Gemini all based on transformers.

The ChatGPT Wrapper Dilemma

It’s worth clarifying the difference between a model like GPT-4 and an application or service like ChatGPT. GPT-4 is the actual LLM or model. You’ve probably never interacted with GPT-4 directly. ChatGPT, on the other hand, is an application that took the world by storm—you and almost everyone you know have likely used it. You input your text or prompt into ChatGPT, which then sends it to the GPT-4 model. The model processes it, generates a response, and sends it back to ChatGPT, which displays it for you.

Now, suppose you don’t like the way ChatGPT looks or functions, and you want to create something that suits your style better. You could design your own web page, with input fields or pre-selected prompts, and then send that input to an API like OpenAI’s (or Google, Anthropic, et al). The API would return a response from the model, just like ChatGPT does, and you could display or use that response however you like. This concept is commonly referred to as a “ChatGPT wrapper.” And it’s not limited to chatbots—you can use this framework to automate all kinds of tasks that could be incredibly useful for your business. Essentially, you combine some subset of data with a pre-determined prompt, and the response becomes business process automation. It’s valuable—so valuable, in fact, that companies worldwide are using these systems today to drive efficiency and reduce costs.

The term “ChatGPT wrapper” often gets a bad rap, and sometimes for good reason. Like the crypto boom during the pandemic, the AI craze has brought out a lot of so-called experts, many of whom rushed to create applications that might not live up to their hype. These products are often pitched as cutting-edge tech employing advanced AI to solve all your problems, but they can be more like vaporware—solutions in search of a problem, with questionable development quality and potential data privacy concerns.

But not all “ChatGPT wrappers” are bad or without value. They just need to be evaluated differently than other types of software. The technology in these applications is generally of little value on its own—it’s simply a framework to process queries through a major LLM API and handle the results. In fact, with a team of average developers and some time, you could probably recreate these results yourself. There’s nothing technically defensible here. The real value comes from how well the application automates specific knowledge or processes unique to a business, often with the help of subject matter experts. We’re seeing companies use these types of applications in areas like customer service automation, content creation and marketing, employee training, call scoring, QA, security compliance, and monitoring—with tremendous results. If the application works as intended, is priced reasonably, and drives efficiency within your organization, it might be worth considering. Just vet it more like a service than traditional software.

Challenges and Limitations of Publicly Available LLMs

Publicly available LLMs offer incredible power but also come with several challenges, mainly due to their lack of transparency and the inability to audit key aspects like training data, source code, and internal processes. These issues are often derived more from the underlying model than to the wrapper application built on top of them.

Data Privacy and Security:  

Using public AI models often involves sending data to external servers for processing, which can be risky if sensitive or proprietary information is involved. There’s potential for data breaches or misuse. This is particularly concerning for businesses in regulated industries like healthcare or finance, where compliance with data protection regulations like GDPR or HIPAA is critical. Enterprises operating in multiple countries may also face data sovereignty issues, where data processed by public AI models might be stored or handled in regions that don’t comply with local data laws, leading to legal risks.

Models for the Masses:  

Public LLMs are designed for broad applicability, which means they might not be perfectly aligned with your business’s specific needs, industry, or brand. While some customization is possible, tailoring the AI to your specific requirements (e.g., incorporating proprietary knowledge or industry-specific nuances) can be challenging without investing in more advanced, costly custom solutions. At the enterprise level, organizations often need tight control over the tools and technologies they use. Public AI models, managed and updated by external providers, can pose challenges in governance, version control, and ensuring the AI behaves as expected in all scenarios. Companies also have little control over updates or changes to the service, which could lead to unforeseen challenges if critical features are modified or phased out.

The Black Box Problem:  

Public LLMs mostly operate as "black boxes," where the underlying algorithms, code, and decision-making processes aren’t open to inspection. This opacity makes it difficult for businesses to understand how the model arrives at its conclusions, which can be problematic in contexts where accountability, explainability, and transparency are critical (e.g., in finance, healthcare, or legal sectors).

The data these models are trained on is generally not fully disclosed. This lack of transparency means businesses can’t verify the quality, relevance, or appropriateness of the data used, which could lead to issues like bias, inaccuracies, or the propagation of incorrect information. Many industries require compliance with strict regulations that demand explainable AI. If you can’t audit the source code, training data, or algorithms of the AI you’re using, you may struggle to demonstrate compliance with these regulations.

If Not Big LLMs, Then What?

Large publicly available LLMs are powerful and valuable in today’s business landscape, but they come with significant limitations and challenges. Given these limitations, it makes sense to explore alternative options. These alternatives usually require more technical expertise and specialized skill sets, but let’s touch on a few.

Fine-Tuning:  

Fine-tuning involves taking a pre-trained model, which has been trained on a vast, general dataset, and further training it on a smaller, more specific dataset relevant to a particular task or domain. This allows businesses to leverage the power of large, sophisticated models while tailoring the outputs to be more relevant, accurate, and aligned with their specific needs. However, while fine-tuning publicly available LLM’s can significantly improve the quality and specificity of the results, most of the other challenges and limitations still apply.

Custom AI Models:  

Developing custom AI models tailored to specific business needs offers complete control over the model’s architecture, algorithms, training data, and functionality. It’s the highest level of customization and can directly address unique challenges in your business. Developing your own AI model means you own the intellectual property, which can be a significant competitive advantage. It provides freedom from third-party dependencies, ensuring your AI capabilities are fully under your control. By designing a model specifically for your application, you can optimize its performance for your particular use case, potentially achieving higher accuracy and efficiency than with a fine-tuned LLM.

But there are downsides. This approach generally requires a team of data scientists, software engineers, and domain experts to design, train, and maintain the model. The process of developing and training custom AI models can be lengthy, often taking months or even years to achieve a production-ready state. This can delay time-to-market for AI-driven products.

Open Source:  

Open-source AI models are AI systems whose code is freely available for anyone to use, modify, and distribute. They promote transparency, allowing users to understand how the models work and ensuring trust in their outputs. Open-source models also enable you to customize them to your specific needs, giving you the best of both worlds—the power of a large model and the flexibility to tailor it to your unique requirements. While fine-tuning an open-source model is significantly faster than building one from scratch, it still requires skilled personnel to implement and adapt it effectively. There’s a robust community of developers and researchers who contribute to the improvement of these models, which means they are constantly evolving and getting better over time. This approach not only democratizes access to advanced AI technologies but also helps in addressing ethical concerns and reducing costs by eliminating licensing fees.

While fine-tuning allows for some level of customization, you are still constrained by the architecture and initial training data of the base model. This can be limiting if your use case requires highly specialized or novel functionality. The pre-trained model may not perform optimally on tasks that are significantly different from those it was initially trained on. Fine-tuning can help, but there may be inherent limitations that cannot be fully overcome. Depending on the origin and nature of the pre-trained model, there may be concerns about data privacy, especially if the model has been trained on data from public sources. Fine-tuning on sensitive or proprietary data requires careful handling to ensure compliance with regulations.

Making Sense of It All

New solutions often bring new problems, and AI is no exception. The rapid advancements in AI have unlocked vast amounts of metadata, presenting companies with unprecedented opportunities—and challenges—in making sense of this information. As businesses navigate this evolving landscape, the choice of LLM will be crucial in turning complex data into actionable insights and maintaining a competitive edge.

The ability to generate meaningful prompts that extract transformational insights from data has rapidly advanced, outpacing our capacity to effectively interpret and apply it. Organizations now face the daunting task of not only accessing this wealth of knowledge but also organizing, analyzing, and applying it in ways that drive real-world impact. This underscores the critical need for robust data management and analytics strategies that can keep pace with the power of modern AI.

The Future

It might sound biased, given my love for open source and the community, but I genuinely believe open source is the way forward in AI development. Open-source models promote innovation, collaboration, and accessibility, allowing the global AI community to accelerate advancements, tackle complex challenges, and democratize AI technology. This approach fosters a more inclusive, transparent, and ethically aligned AI landscape. By empowering everyone to participate in AI development, open source ensures that the technology evolves in a way that benefits all, creating a more equitable and advanced AI future.