Before You Build Another AI Feature, Run This Test

Most AI initiatives don't fail because the model wasn't good enough. They fail because the idea should never have made the roadmap.

You've probably seen the meeting. Someone asks, “Where could we add AI?” Twenty minutes later, the whiteboard is full: copilots, agents, assistants, summaries, recommendations. Engineering sees possibilities. Sales recognizes a few customer requests. The CEO has a favorite. Everyone leaves feeling like progress was made.

Six months later, three ideas have shipped. Customers barely use them. Nobody will pay more for them. Finance asks what they actually contributed. And somehow the conclusion is: we need better execution.

I think that's usually the wrong diagnosis.

The mistake happened before engineering wrote a line of code. We asked, “Where could AI go?” instead of, “Where would AI create enough value to deserve investment?”

That difference sounds small, but it isn't. One question generates ideas. The other allocates capital.

When I look at AI opportunities in an existing product, I want four questions answered before I get excited about the technology.

Start with economics, not AI. Look for work customers or employees already spend meaningful time or money doing repeatedly: manual analysis, data entry, research, configuration, reconciliation, document preparation, repetitive decisions.

The clearer the existing cost, the easier it becomes to understand what improvement is worth. “I can add an AI assistant here” isn't a business case. “This takes every analyst six hours per week” is much closer to one.

Now ask what your product knows that a generic AI tool doesn't. That might be proprietary data, workflow context, historical decisions, customer configuration, domain expertise, or simply a privileged position inside an important workflow.

The goal isn't to sprinkle customer data on an LLM and call it a moat. The question is whether you can produce an outcome here that would be difficult for the customer to get by opening ChatGPT, Claude, or another generic AI tool in a separate tab.

If the answer is no, you may still have a useful feature. But don't confuse usefulness with defensibility.

This one eliminates more ideas than people expect. If the AI works, what changes? Revenue? Conversion? Time-to-complete? Cost-to-serve? Retention? Error rate? Throughput?

If the answer is “customers will love it,” keep going.

You don't need perfect attribution, but you do need a result you can observe well enough to decide whether the investment worked. Otherwise, you'll ship the feature, announce it, put “AI-powered” on the website, and still be unable to answer the CFO's simplest question: What did we get for the money?

This is the filter teams underestimate most.

AI will be wrong sometimes. The strategic question isn't whether failure exists. It's what failure costs.

Can the user spot the mistake? Can they reverse it? Can a human review it before anything consequential happens? Does the error create five seconds of annoyance, or does it create a financial, compliance, safety, or customer-trust problem?

This doesn't mean AI only belongs in trivial workflows. It means the economics of failure belong in the business case.

A draft can be wrong and corrected. A recommendation can be ignored. An irreversible action needs a very different standard.

Customer support is a useful example because at first glance, it looks perfect for AI. It's expensive, repetitive, and high-volume. That makes the savings opportunity obvious.

Klarna became one of the most visible examples. Its AI assistant handled millions of conversations and the company reported significant cost savings. But Klarna's experience also exposed the other side of the equation: there are customer interactions where access to a human still matters, particularly when the problem is complex, ambiguous, or emotionally charged.

The lesson isn't that AI customer support doesn't work. It clearly can.

The lesson is that passing the cost filter isn't enough.

A product leader also has to understand what happens when the AI encounters the part of the workflow where judgment, trust, or ambiguity matters. That's a very different question from “How many tickets can we automate?”

Imagine you have 40 AI ideas. Don't rank all 40.

Instead, run them through four questions:

Is there expensive work here?
Do we have an advantage?
Can we measure what changes?
Can we contain the cost of being wrong?

Some ideas will disappear immediately. Good.

A few will look useful but easy to copy. Others will have measurable value but unacceptable failure economics. Maybe two or three survive everything.

That's not a failure of ideation. That's strategy.

Because the scarce skill in AI isn't generating possibilities anymore. Everyone can generate possibilities. Your CEO can generate another ten AI ideas before lunch. Your competitors can do the same. So can every PM on your team.

The scarce skill is deciding which opportunities deserve engineers, budget, and organizational attention, and which ones don't.

And there is a second skill that matters even more: you have to defend the cut.

Killing idea #17 is easy in a spreadsheet. It's much harder when the CEO loves it, Sales has already mentioned it to a customer, or Engineering has built a prototype everyone thinks is cool.

That's where product leadership moves beyond prioritization. You need to be able to walk into the room and say: “We're not funding this. Here's why. Here's the evidence. And here's what we're funding instead.”

That is AI strategy.

Not the brainstorming. The decision.

Hit reply and tell me. I'm genuinely curious which of the four filters it fails.

If making the decision isn't your problem, but getting executives to support it is, that's exactly what we'll practice in my next live workshop:

Four hours of hands-on work on a real proposal or executive update from your own job. You'll learn how to translate a product decision into the language executives use to allocate capital, assess risk, and make the call, and you'll leave with a rewritten proposal you can actually use.

Next workshop: August 14