Why AI projects often fail…and how you can solve that

Our CEO Davio Larnout shares his insights on why AI projects go wrong, gathered during five years of working with organisations in many different industries. In this interview, we asked him a few questions to show you the value you can gain when truly integrating AI into your organisation’s processes. 

Q: We still hear the phrase “AI is a hype”…is that true? 

D: Over the years, we hear fewer and fewer people saying that, which is great. However, AI is still not seen as a strategic technology in some cases. We noticed that many of our customers or potential customers want to work with AI, but it is still treated as a checkbox exercise, and when you do AI for AI, it’s a recipe for disaster. 

Q: What would AI's impact, if not treated as a checkbox exercise? 

D: It’s interesting. Our research on our customers shows that AI contributes up to 25% of their bottom line. We’re not alone: McKinsey believes it’s about 27% of the bottom line. We see this at our customers, who are the leaders in their field, and even they are accelerating their efforts at scaling AI across their organization. They know that it will give them a strategic advantage.

Q: Could you give an example? 

D: Of course. To better understand this, think of a factory that does everything by hand. If electricity gets introduced, they can start doing experiments, so they introduce a power drill at one workstation. 

Suddenly, this workstation can produce 5, 10 times the previous output. Great, but the impact at the factory level will be marginal. Almost nothing has changed, just that one workstation. 

When you start embedding electricity across the organization in every workstation, it starts generating a strategic impact, which is the same for AI. It's a technology you need to scale across your organization, which means that you will fail at some point or another. At some point, some things will work, and others won't, but you need to keep investing in it. And that's what we see in our customers leading the way today. They are achieving 25% of their bottom line. 

But they are also taking a head start right now, and they are accelerating thanks to the efforts of scaling AI across their organization. That is what brings them true strategic advantage. 

Q: So besides not scaling AI at all levels, what is something else that pushes organizations back in their AI efforts? 

D: We see that accuracy is still everything for many organizations. This is a big theme in AI, and in some cases, when our customers start talking with us, they expect 99% accuracy in the end solution. 

To step back, I mean the accuracy of the model prediction/recognition/etc. For example, detecting something in an image, let’s say, crops in satellite images: it would mean the solution detects 99% of them correctly. 

At Superlinear, we believe accuracy is important, but it’s not everything. If you only focus on that, it’s a recipe for disaster. Instead, we have identified seven dimensions of success with an AI project. 

Q: Could you clarify these dimensions? 

D: Sure. I’ll go deeper into two of them today. One of them is fairness. I’ll give an example of a project we did with one of our customers, where we helped candidates find a job with AI, matching CVs and vacancies. 

It was one of our first projects, and we wanted maximum accuracy, so we went in with deep learning. If you optimize for accuracy, the solution will detect patterns, which can become problematic. For example, you have a job description of a cleaning lady that matches people with Swahili in their CV, as there was a pattern in the data. Now, we don’t want that, of course. 

So, we had to sacrifice accuracy to achieve a fair solution. It’s a trade-off, so we avoid having a discriminating system. That also means going back to the drawing board because you have to choose other algorithms. You have to rework your data. 

To avoid going through the whole cycle again, addressing fairness at the beginning is essential. 

Q: Very interesting. What is the second dimension you wanted to focus on? 

D: The second dimension focuses on operations, commonly referred to as MLOps. Something, still in the conversation today, is that the more data, the better. In fact, you could say the more data, the better algorithm, so better accuracy. But then, it's not true if you look at the business case. 

That’s because more data means more storage, more compute cost and not necessarily more value either. I will refer back to my example above of matching jobs with CVs. 

As we acquired customers using the solution, we could tell that the additional data was becoming redundant: there was a lot of duplication. 

The value of data is not in the volume; it’s in having the right data. That means asking yourself what data points allow the algorithm to learn from? Then, and only then, your data is valuable. The rest will cost you money because you need to store it, process it, etc. 

You’ll save money by thinking about the data you need. For this one, we have a trick: the data flywheel.

Here, you compare the data you add with the data you already have, and you only add what adds value to your algorithm. It’s easy to explain, not so easy to do! But you need to do it upfront; otherwise, your costs will explode. 

Q: Great. So we covered why AI is not a hype and accuracy is not everything. Is there anything else that organizations should consider? 

D: I want to mention again you need to consider AI as a strategy. You need to invest in it, and scale it across your organization. One killer use case doesn’t exist. Only when you embed AI across your organization in multiple processes will you leverage its value. In addition, accuracy is not everything, and you need to look at different dimensions. 

You also need to consider the end user if you want your AI project to be actually used. 

How you deal with an AI project is also where we see things going wrong. With AI, many things are coming up all the time. That means that at every sprint, you need to check what’s in the data, what it will cost us, is the data fair? After you answer these questions, you can decide how to move forward. 

Q: What is the best way of deciding how to move forward? 

D: What is very important is to decide from the start what you want to achieve with your AI project. With an AI project, you need to optimize a certain part of the organization and bring the business along with it. When you have decided what you want to do, then you can easily navigate the project and know why you are doing it. 

Along the way, you can then measure the impact the project has. How does every sprint impact the main objective? 

In addition, as I mentioned before, you need to involve the end user. It still happens now that organizations don’t do that and incur a huge loss of time and money. For example, a few weeks ago, I heard a CIO saying they spent €600,000 on a solution to support their consultants, and they didn’t want to use it. That’s because they were not involved from the start and didn’t see the value of using the solution. 

At GSK, we did a project on a certain aspect of vaccine development, and we involved the lab technicians from the start, asking them questions like: “How is this going to make your life better?”. We ended up with engaged users who became the number one ambassadors towards their colleagues. That is what you need if you want to scale with AI. You need ambassadors; you need people that want to use that new technology. 

Q: How did you capture feedback from the end users? 

D: We showed them a working solution from the start. You can iterate around it and capture feedback when you show the value very quickly. Thanks to open-source AI, in a few weeks, typically six, you can quickly build a working solution. 

This solution won’t have everything but the core value of what you’re trying to achieve. In six weeks, you build the first solution, start capturing feedback, and only then will you have a successful AI solution. 

You then move into agile development, capture more feedback, adopt your solution and consider the seven dimensions I mentioned above. 

Q: Thank you very much, Davio, for your insights!

Want to know more about the seven dimensions of a successful AI project? Then book a call with our Account Executive Tomas below ⬇️