Working with Public Cloud the last 15 years, it has always boiled down to a new way of working (and not always the technology). Adopting a new agile approach to building and delivering services and adopting a common cloud framework from the cloud providers, but in the center of this is the CCoE (Cloud Center of Excellence) which helps an IT organization become a partner to the business and helps to maintain balance and support, hence I like this illustration from Microsoft. Where many IT departments today are still more on the left side, where IT is a centralized unit (typically under HR or Finance) but most need to understand that IT is a core capabilities for most businesses today and should be seen as an enabler for the business and not just a “function”.
While the journey has taken a while for most organizations to adopt, I have luckily been working more and more with companies now that have this structure up and running. Of course depending on the size of the company, the CCoE can be anywhere from 3 people to 20+ people that have different roles and responsibilities, but the crucial part is that the business goes to the CCoE when they have a new initiative or project they need to deliver.
While the cloud journey has going on for many years now, all of a sudden here comes the AI train. Where development is happening so fast it is extremely hard to keep up.
And while I myself have been working hard to keep track on the updates from the cloud providers, I can safely say that AI is moving ALOT FASTER, because all the technology providers are working on it. We also have so many new startups because AI has now become to easily available, and of course all the ISVs are now looking into how they can embed AI into their existing ecosystem.
What I am seeing now is that many companies have many AI initiatives coming from different parts of the business and many people inside the company that are really eager to start to use it and jump aboard and use ChatGPT/Gemini/DeepSeek/Copilot or whatever is available to them.
Now we have the cloud journey all over again, but in a different form… Where IT or the CCoE is unsure how to handle AI into the mix, since it can be a mix of so many different elements.
- PaaS services and LLM infrastructure
- Copilot 365 / Gemini
- LLM Security
- Agent Ecosystem
- AI and data processing
- Coding Assistants (Github Copilot, Cursor)
- Service Now – Now Assist, Copilot Studio, Google AgentForce, Langgraph, Elevenlabs
Suddenly we now have a huge ecosystem of different tools and services that can be from a cloud providers or running locally on our own infrastructure. We have tools that our developers and regular users have access to with Copilots. We need to understand the legal ramifications and what kind of data these services actually use. Suddenly we now are getting bombarded with different use-cases from the business that wasn’t there before.
Now the big question is how should companies handle all the AI use-cases, initiatives and projects? should they build a AI Center of Excellence? Should it co-exist with the CCoE? Do we need one?
To be blunt, not everyone needs an AI Center of Excellence, some or just better off with starting with some common guidelines on how your company should use AI. Sometimes people just want a set of directions.This is what I call guided exploration, just having some policies in place to say that
- AI Should be used only for certain sets of data and the following tools can be used.
For larger organizations where more use-cases can be identified, you should have more training and education in place to give people in the organization a better understanding on what AI is and how they can use it. This will also drive up the amount of use-cases and initiatives within company because more users now see the potential.
However if you have a larger IT department you should have a dedicated team that handles AI initiatives within the company.
Why? I have already seen larger companies have so many different and similar AI initiatives from different parts of the business. Like 3 similar RAG projects, and 4 different agent ecosystem exploration projects in the SAME company. Think about how much time is spent now from different teams in the same company trying to reach the same goal. Also the business might have so many different use-cases, how should they be handled? By the business themselves, because IT does not wanna touch AI?
When building an AI CoE it should have the following
- Have executive sponsorship for authority, budget, and influence.
- Appoint a AI leader to drive strategy and ensure alignment. This should be an individual that has strong technical competency but is also able to communicate with the business.
- Build a multidisciplinary team covering business, data, AI, and governance roles.
- Integrate with existing teams, like the Cloud CoE, to avoid duplication and silos. In many cases AI initiatives overlap with the CCoE.
And it should also have a funnel that allows the business to pitch use-cases and ideas they have regarding the use of AI. If a team wants to look at the use of Cursor or Github Copilot, the AI CoE should guide and educate them in how they should use it but also measure the effectiveness of the tool. If a team wants to have a AI tool that can handle speech-to-text for transcription the AI CoE should look into which LLM or model or service provides the best capabilities for that specific use-case.
This can be a Microsoft Forms or a DevOps User Stories that allows the users to pitch ideas or specific use-cases they need solved.
So to answer the big question, do we need an AI center of excellence?
Based upon my experience, if the company is of a certain size I already see so many AI initiatives coming from the business (and many of those are so similar in shape and form) that have a AI CoE in place would save the time and effort spent, while also ensuring that they can see the wider picture on how a tool or product can benefit other teams as well) However it requires a strong technical team to handle this, since this is a broad field and is always developing.