The Future of Enterprise AI: Insights from Robbie Jerrom on Open Source Solutions
Wed 19 Feb 2025
Artificial intelligence is transforming enterprise technology, and open source innovation is at the heart of this shift. In this sponsored interview, Principal Technologist for AI at Red Hat, Robbie Jerrom, shares insights on the evolving AI landscape, the role of open source in enterprise adoption, and how businesses can navigate challenges in security, scalability, and hybrid cloud integration.
– – – – – –
How have you seen the landscape of enterprise AI and open source innovation change?
Great question. It has accelerated massively over the last two years, but we have been seeing the embrace of open source growing over the last 30 years. When we started with Linux and open source many years ago, we were kind of the outliers. No one really understood how we were going to contribute to free software or how it was going to work.
But with the explosion of cloud computing and the realisation that everyone can see the software, trust it, and add their own value to it, innovation has become quicker and easier. That collaborative effort has continued into AI, open source AI, and open data.
It has accelerated because, similar to the early days when we could examine and understand how Linux kernels were built, we can now do the same with AI. By looking at an open-source AI model and its training data, we can better assess how much we can trust it.
If we want to contribute to the model from a knowledge perspective, we can. For example, if I know something the model does not, I can say, “Hey, this is how this widget works,” and provide that knowledge to the system. You cannot do that with closed systems, you can only do it with open source.
That is why open source is growing and it is getting bigger, better, and more exciting. With AI, this openness is critically important to enterprises. You have to be able to trust the model that is answering questions for your customers, your business, or your insights. That is critical. If you cannot see it and understand it, it is hard to trust it.
How do you see open source tools and platforms shaking the shaping the future of this scalable AI?
Once again, it is the same play as with Linux. We are already seeing open source tools and frameworks from Meta and others, such as LlamaIndex, IBM Granite Models, and InstructLab, providing tools for enterprises to get started easily.
There used to be a steep learning curve to get into machine learning and AI. The stacks were complicated and difficult to use. With these new open source tools, I can download InstructLab and start on my laptop, learning how to build a model or perform tasks at a small scale, whether I am a developer using a modelling application or adding knowledge to a model.
Then, I can take that same toolset into the enterprise and run it on a Kubernetes platform in my data centre or in the cloud. This consistent experience is driving rapid innovation.
It is making the tools accessible for someone who wants to learn, and also for scaling up, and running a full team while maintaining that consistent experience. That is what we are seeing with open source.
You can run it on a gaming GPU, and then scale it up to the cloud or a data centre with clusters. It is the same hands-on experience, so the learning curve has come down with open source.
How can open source AI allow organisations to innovate, and what strategies can address common concerns about security and support?
The innovation piece is all around the tools and just thinking about what you might be able to do with AI as a business.
And that is something we are quite familiar with now. Individuals have played with things like ChatGPT or Copilot, and we are getting a feel for how AI can work day-to-day for us. That maps across to the enterprise. How can I deliver a better experience?
But then there are concerns around security, supportability, and certainly privacy. Data privacy is a huge one for enterprises. What we are seeing is organisations are reshaping a bit to have an AI Centre of Excellence or sometimes an AI council or board.
These groups consist not just of technologists and developers, but people from HR, legal, and other lines of business. Those leaders are discussing how their organisation should embrace AI, what policies they should have around the ethical use of AI, and when it should not be used, especially with European regulations around data privacy.
Those experts, who are sometimes enthusiastic people, help guide enterprises on how they can best take advantage of AI through their enthusiasm and innovative spirit. Before we even get to the technology, it is about asking “Could we?” and “Should we?” before we start throwing lots of resources at it.
That is the best way to adopt AI. It helps address security concerns, support concerns, and data ethics before spending money on resources and focusing developers and other staff on these initiatives. That is where we have seen enterprises and organisations move forward quickly: Start with “Should we?” and “How could we?” and then move on from there.
Private generative AI is increasingly being explored by enterprises. What unique opportunities does it offer, and how can businesses balance innovation with data privacy?
I am a strong advocate for privacy, especially data security for individuals and organisations.
That is what is driving many of the private generative AI conversations. While these incredible foundational models know a lot about various topics, they do not understand how an enterprise operates, how my organisation, my development team, or my internal projects work. The unique internal knowledge that sets my company apart simply does not exist in these models.
I want to take advantage of generative AI, but the models need to understand my terms, how I speak, and what is appropriate to communicate to my customers. This includes both language and knowledge.
Being able to take a foundational model and teach it, similar to onboarding a new hire, is key. This process involves explaining how we do things, how our products work, and how our development processes operate. It also introduces terms and phrases that may be unfamiliar because they are unique to us.
Training that into a foundational model and connecting it to company data makes the AI useful. Now, I can scale it and I can make the individual I have onboarded useful across the enterprise. I can deploy it in a private data centre or internal system. Once I am confident, I can push it out to a hybrid environment and scale it up to work with any cloud and customer-facing systems.
We have seen some serious mistakes in the past, such as companies releasing AI chatbots into customer-facing environments without providing proper training, security, or knowledge. This is how we ended up with situations like people negotiating car deals for a dollar.
That privacy, learning, and teaching the AI how we do things is critical to success. Ultimately, take the AI, make it yours, help it understand what makes your company special. From there you can start using it. That way, it is not going to sharethings that you do not want associated with your organisation.
What are the most important considerations when building AI platforms for hybrid environments?
The most common challenge is consistency. We have a tremendous amount of choice when it comes to AI services and capabilities.
Each hybrid cloud environment and hyperscaler offers different capabilities. They are all fantastic, but they are different for each cloud. This introduces complexity in an already complex environment.
If the models are different, they need to be trained, tested and iterated differently, which leads to different responses. That can be a challenge for organisations with multi-cloud or hybrid cloud policies, or those that have part of their strategy in private cloud.
The consistency of a foundational model that we have trained on our enterprise data and connected to that data, is crucial. Otherwise, if I ask a question and the response from one hyperscaler differs, it could confuse my customers or internal users.
The key is consistency, consistency of user experience, for the IT ops teams managing the platforms, and in model choice. Going back to open source models, we can build, see how they respond, and control them.
If you want to move to a new and better model, you can do it on your terms after evaluating it, rather than relying on an automatic upgrade. This is a major concern and focus for enterprises, on how to ensure a consistent AI experience.
This can be achieved with hybrid cloud, but you must build a consistent platform on top of the hybrid cloud to make it work.
How can organisations navigate the complexities of adopting AI and hybrid cloud environments to achieve their business goals?
The first step is starting small with a business goal. When getting started, do not try to boil the ocean on day one. Create a small, internal service, which helps you adopt AI. It is a way to learn how to use a model that is connected to your data, and success grows from there.
One of the biggest challenges enterprises face when adopting a hybrid approach to cloud or AI is proving that first use case. Once that is proven, it grows from there. If one line of business has a chatbot or a help support system that works and is cool, it becomes well-known. Other lines of business want to know how it was achieved, generating a centre of excellence and buzz around the technology.
That is how you start to scale it. With hybrid cloud, it is about consistency. Push the solution out across the organisation to demonstrate its value.
Ultimately, the key is to start small with a business goal. Do not go in saying, “We are going to AI the world.” We have seen the most success with small steps, learning how to do something, and then growing it from there. It is the same approach we took with hybrid cloud: start with small applications across multiple clouds, learn how to do it, and then move on to the larger projects.
What are the biggest challenges in operationalising AI at scale and how can businesses overcome them to achieve sustained value?
The biggest challenge when operationalising AI are cost concerns, particularly the cost of running AI services at scale, which involves using many GPUs and cloud resources. Even services that charge per request can become expensive as usage grows, so controlling costs is a key element.
Organisations that are succeeding are being forward-thinking. They are asking themselves, “Do we need an AI model that knows all of the internet’s knowledge and can answer questions about anything, or should we focus on a smaller model that does not cost as much to run?” With a smaller model, you can train it on your data, which allows you to maintain control. This aligns with the idea of “take it, make it yours.”
For example, smaller models are capable enough for specific tasks, like a support chatbot. It does not need to know everything, it just needs to understand your product, support processes, and internal knowledge. As long as the model understands the languages you and your customers speak and is connected to your data, smaller models are much easier to train, run, and host.
We are seeing this control point in action, with small models being used for specific purposes like chatbots or helping developers code in languages like Perl or Python. For example, one customer has a unique internal programming language that does not exist online, and they are exploring how to train a model to understand and use it to assist their developers.
Smaller models require fewer resources to run, such as a single GPU instead of multiple GPUs, which makes them a cost-effective option. Additionally, smaller models allow for better fine-tuning of user experiences, giving organisations more control over how they are used.
As AI adoption grows, how can organisations ensure their approaches are ethical, responsible and sustainable?
We have discussed the concept of an AI Council or board, which is a critical element in ensuring the ethics and responsibility surrounding AI. In Europe, we are seeing more regulation around what is acceptable AI usage, what data we can use as UK or European citizens, and where that data can be processed. The legal frameworks are catching up, but they often lag behind the technology. However, we are a bit more on the pulse when it comes to AI legislation.
Having an ethical Council in place, which sets clear rules and guidelines, is essential. This is typically documented and outlines what is acceptable within the organisation, much like a company strategy. For example, when a customer or a regulator asks about your position on AI, you will have a written policy in place.
This policy should cover the use of AI both for customer data and products, as well as for employees. It is important to define when AI can be used, like in marketing or for writing proprietary code. Should employees use AI chatbots or coding assistance for these tasks? These policies help ensure ethical and responsible AI use.
On the sustainability front, the challenge comes from the power consumption of GPUs. However, we are seeing improvements as we become more efficient at building generative AI models and managing their architectures. Running smaller, task-focused models that are open source and can be self-trained costs less to run. By using small models, we have more control over sustainability, as we know exactly how much power they consume, compared to large models that aim to know everything.
There is always room for more progress in sustainability, but by using open source models and optimising them as a community, we are taking a step in the right direction. This allows us to run more efficient code on smaller GPUs and get more out of it.
What industries do you believe are set to see the most transformative impact from AI and open source innovation?
It is a tough question. I once asked a friend of mine at NVIDIA a similar question, and his response was, “Tell me the best use cases for electricity, and we will go from there.” That made me pause because AI is impacting every industry, from agriculture and farming to technology sectors and more.
We are seeing its effects across the board, whether it is using AI to analyse existing data or completely transforming how we do business. I believe the open source aspect will be critical, and we are seeing organisations that traditionally have not been involved in open source becoming more engaged because it is so intriguing.
There is not a single industry where we will not see an impact. We are seeing commonalities across industries. For example, Farmers now have massive amounts of data and AI is helping them turn that data into something exciting. They can predict farm performance over the next 12 months, which is crucial to their success.
Similarly, industries with abundant data, like energy, are also benefiting from AI advancements. Other unexpected sectors, like civil engineering, are also being transformed. In the past, assessing the quality of concrete in buildings involved taking lots of photos, but now drones can fly over and highlight flaws in the concrete, making the process much more efficient.
AI is everywhere, and innovation is accelerating. Every time I look, there’s a new way of doing something: whether it is writing code, analysing music, optimising video, or understanding patterns in concrete or soil. It is an exciting time.
What advice would you give to organisations just beginning their journey with AI and hybrid cloud technologies?
Start small with a focused team of individuals who are exploring what is out there. The most success we have seen comes when this group includes people from different lines of business who understand more than just the technology. For example, those who know about hybrid cloud, tractors, communications, or concrete. These individuals provide valuable insights into how AI can be applied.
Technologists can explain how AI works, but business experts ask the most compelling questions. They are the ones who ask, “Can AI understand an image of concrete and identify if it is broken?” or “Can it help me write code that does not exist anywhere else?” These questions drive real use cases.
This is where excellence begins, by starting small with clear, targeted goals. The hybrid cloud approach can be a great way to quickly onboard and test things while iterating. But there is always the risk of getting locked into a specific cloud service, which ties back to the open source concept. That should be part of the strategic guidance. Whether your organisation is open source or not it should be a key consideration in the planning.
