AI go to market hasn't really changed

I’ve been spending a lot of time with AI companies. Some I’m getting demos from, others are looking for help: customer or product development, sales, marketing–there’s a lot to do.

I see the same exact patterns repeating themselves that I saw in 2015. While large language model (LLM) technology is a massive leap from what we were able to do back then, the same GTM challenges remain. Here’s what they are and how we solved them the first time.

Expectations

The first is the expectations gap. I get my shiny new AI tool and ask it what I think is a simple question. It doesn’t answer it correctly, or at all. I’m disappointed and I churn.

This expectations gap goes all the way back to IBM Watson, the first widely advertised AI solution. Built to compete on Jeopardy, in the televised contest Watson said that Toronto was the US city with a couple of airports with links to the two world wars. Obviously–obviously!–Toronto is not a US city. Or is it? It’s all about expectations. The way Watson was architected, and the way the Jeopardy data was fed in, made Toronto its low-confidence first answer choice (Chicago, the correct answer, was second).

In Perfect Price’s time, the expectation gap took the form of disagreeing with prices. If we generated 10,000 prices, and the user saw 5 that were provably wrong, they would be angry and not use the software. Even though there’s no world in which a human could achieve 99.95% accuracy, humans still thought the AI should be accurate. So you were disappointed; it didn’t meet expectations. And churn.

We found two solutions to this. First, we constrained the problem. The sales process and the UI of our application became very aggressive about setting expectations. We invested heavily in rules both behind the scenes, to avoid even the 0.05% of “wrong” prices.

Second, we enabled humans to take control with basic rules–thus changing the expectations entirely. The AI went from being the hedge fund managing your business to make more money to more of a workflow tool that augments your team. You think we are charging 10% too much? You’re a human and you must be right; here’s the button you push to lower by 10%.

The principle here was focusing on the customer. Early on, we thought our customers were simply wrong (or worse). But as we grew as founders, and learned more, we realized that even if they were wrong–if we wanted them to stay customers, we had to address their concerns in a way that was intuitive to them.

Understanding

We thought selling dynamic pricing software would be like selling free money. Use it, make more money. That’s how it went in adtech at Drawbridge–why would it be more complicated for pricing? But humans want to understand, and prices are closer to their business and more tangible than tens of millions of ad impressions spread around the internet. This meant we, or rather the AI, had to explain itself. If the AI says a price should be $500, why? There has to be a reason!

We looked at it a little like a hedge fund. Renaissance Technologies is the most successful investment manager in history because nobody there cares why a company or currency went up or down in value. They just care if their predictions about the change are accurate more than 50% of the time. Turns out, there’s a reason why successes like RenTech are so rare–humans crave understanding. And they let that limit them.

Our AI couldn’t explain itself. That was a much harder problem. Without LLMs, it was totally unsolvable. We could prove that our model was better than humans–but we couldn’t tell anyone why it did what it did, or even why it was better on a fundamental level.

The solution we found most suited to this was simply to provide reasons anyway. For example, if the model suggested a higher price on Saturday night for a home rental in Hollywood, California, the human would want to know why. Let’s say, for the sake of this example, Maroon 5 was playing a concert at the Hollywood Bowl that night. We showed a feed of local events, and the human would see the Maroon 5 concert and agree “Yeah, there’s a big concert so prices should be high.” The human has their reason, and feels confident accepting the suggested price.

But it was a lie! The model, actually, was completely ignorant of that event. The “reason” has, in fact, no place in the model. Why? Because the “reason” a human would accept is actually detrimental to the model.

There’s a concert at the Hollywood Bowl every single Saturday night. And they’re always big shows. The performers only play 1 Saturday night per year, at most, because it’s a stop on a worldwide tour. If they play 2 nights in the same tour, they’re back to back (say, Friday and Saturday). If they even played a Saturday night before, it was probably more than a year earlier and also a different time of year, which introduces a fantastic amount of noise. Demand for an artist changes drastically over years, and demand for attendance at outdoor venues varies by the season–summer, winter, etc. So there’s almost 100% noise and 0 signal from knowing who is playing at the Hollywood Bowl. So the model doesn’t use it. But that doesn’t make that any less valuable or convincing to the human. Because the human thinks the model should know who is playing at the Hollywood Bowl because the human thinks (without evidence) that has something to do with demand for rooms in Hollywood.

We found that by using tactics like this–presenting data to the human as a reason, even if it wasn’t incorporated into the model–increased confidence and helped perceived understanding. We even built small models that were only used for user interface purposes. For example, occupancy is above average, or supply is below average. Competitors are charging more than average. This also fit the job to be done, for the human. If the UI didn’t provide this understanding, they might need to go figure it out elsewhere. By providing it we were helping them do their job. It also made it easier to sell.

LLMs are, by their nature, better at explaining themselves than the type of models we used. But the fundamental principle remains the same. Humans want to feel like they understand why the AI is doing what it is doing. Even if they don’t, actually, understand it, the perception of understanding aides in AI adoption, use and retention. Part of the job to be done is being able to explain yourself as a human to other humans. Even if you are relying on a black box AI tool, you’re still required to “show your work” for some problems. For others (like serving ads), nobody asks too many questions.

Business problem

Being great at the technology does not mean that you have a deep understanding of the business in need of that technology. I think this is the greatest risk facing many AI startups: they’re building something nobody wants or needs.

I’ve talked to several very hot startups today who also have very high churn–70-90%. It’s a large enough sample size to conclude that this is a widespread problem amongst even well funded AI startups. Obviously that’s unsustainable, but if you’ve closed a $25 or $50 million Series A, you can actually sustain it for quite some time. The press will still love you. Prospective employees will still flock to you. But the reckoning will come eventually.

My biggest mistake at Perfect Price was being too focused on the technology, without stepping back and looking at the business problems we were solving. It seemed obvious–manage prices so companies make more money. But when we dug deeper, we learned customer needs were much more complicated. And some didn’t even care about making more money.

The lesson learned was to focus on the customer (and to spend time finding the best customers). Like any software product, ours needed to enable jobs to be done, and could extract value based on the value that doing them successfully creates. Some jobs to be done are obvious, and the value they create is obvious. For many industries and use cases, however, it’s hard to discover this value without knowing a lot about the industry, or asking a lot of questions of potential customers. Don’t skip this step. And even if you skipped it, it’s never too late to go back and do it properly.

Proof of value

A frequent complaint of AI companies today is that they create a lot of value but nobody believes them during the sales process. We had this at Perfect Price in spades. We did a backtest for a major, global company, in which we demonstrated in a scientifically defensible study led by their very competent data scientists that we could make them $1 billion a year. A couple weeks later, they decided not to move forward with a pilot.

This was a company-making enterprise deal valued at millions, and to lose it after what seemed to be a successful proof of value was beyond disappointing.

I asked the business decision maker and our champion, “Did you or the team not believe that we could make you $1 billion?” To my total surprise, he disagreed. “We might not have believed $1 billion, but we were completely sure you could make us $500 million.”

Wait, what? There’s $500 million sitting on the sidewalk and instead of bending down to pick it up, you’ll just leave it there? It came down to the fact that the risk of changing how they did things wasn’t worth it to them.

So they left it there.

Today buyers are saturated with messaging on how great this or that product is, how much money (or other impact) it made for so-and-so. Sales people at AI companies wonder how their product can be so great and have such a big impact for their customers, but nobody new wants to try it out. Turns out, that’s normal.

The solution we focused on was both messaging and finding a market with a burning need. For messaging, we moved away from AI towards the problems and pains we solved. We talked in the customers’ language, rather than about AI, and that created more empathy and understanding. It didn’t look like some fancy magical tool; it looked like a tool they could use to solve problems they had right now.

We also refocused where we sold our product. Everyone wanted to meet with an AI company. And even today, lots of AI solutions are technologies in search of a problem. We started turning down meetings, and focusing on just one or two industries. A more constrained problem, in the language familiar to the industry, resulted in much faster sales.

Strategic value

Our original thesis with Perfect Price was to help price marketplace items. This problem lent itself very well to machine learning, and Uber (among others) had proven how big an impact pricing accurately as a marketplace could have on profitability.

The challenge–which we were unable to overcome–was that these places at that time (2014-2016) didn’t care about profitability. In fact, they explicitly had deprioritized profitability because all investors cared about was growth.

PROS, the public pricing company, had achieved massive sales success in the 2008 financial crisis because companies had cut all they costs they could cut and CEOs and CFOs needed to find something to drive continued profit gains. We were in the opposite market, where everything was go-go-go, and trifles like profitability or contribution margin were considered by management teams a distraction from their strategic goals, of top line growth. One founder even said directly that a price optimization was already priced into their Series C valuation. It was like a hot housing market–the tear downs were being priced like they had already been torn down and rebuilt.

If there’s any lesson here its to really understand the key strategic goals of your customers. If they don’t align with your product, you have a real problem, and at the very least need to find a different kind of customer with different motivations or externalities. With a large enough marketing budget, you might be able to change the sentiment–but don’t bet your company on it.

Human computer interaction design

ChatGPT made the chat interface the interface of choice for AI companies. Being a legit AI company kind of means looking like ChatGPT. But that ignores the customer’s needs and preferences, and the jobs to be done. Chat interfaces have their place, but for most products they have significant drawbacks.

Back in 2017 or 2018 Youngin wanted to build a chat interface and we ended up making “otto” into our homepage (we’re visionaries, what can I say?). Prospects could chat with it and then if qualified would route to sales. The technology wasn’t anything close to what it is available today, but it was an interesting interaction and I think it cost about $50 to run on AWS. However it was obvious that interacting with a revenue management system through chat was a non-starter. How do you change 1,000 prices at once in a chat interface, and check to make sure at least some of them were changed the way you wanted?

Chat creates amazing new HCI possibilities. For example, using Cursor you can upload a screenshot of the console and Cursor will figure out what went wrong on its own–just like if you sent a screenshot of the console to another developer to get help. It’s really amazing. But I only knew you could do that because I saw someone else do it on a tutorial video. I’d never have thought to try without seeing that video, off the platform.

A mix of chat and good old fashioned buttons is how the majority of us will learn to interact successfully with AI products. The AI products that build AI into workflow tools, solving important jobs to be done with high business value, will be the biggest winners. Those obsessed with the “elegance” of a chat interface will see their customers be confused, not know how to use the tools, fail to get value, and churn. Like, apparently, many are seeing today.

How to prevent repeating history

AI products today are a significant leap ahead of the late 2010s. They do things we used to joke about in brainstorming sessions–and do them well. But many sales and marketing challenges remain the same. If anything, the hype is even greater now–which makes it even easier to hide or miss broken product market fit and go to market. And harder to meet expectations.

To any AI founder or seed investor reading this, I’d encourage you to dig deep on the actual business. You can even call it “founder mode” if you want (though I have issues). Is retention a problem? Are users engaging with the product and doing the high-value, key tasks quickly and easily? Can you articulate your customers’ pain points or problems in their own words? Are these critical problems that need fixing immediately, or are they “found money” that will be there in a few quarters, whenever they get around to it? Is your team clear on who are not your customers, and do they feel comfortable passing on meetings? This an many more questions are outlined in our GTM Assessment

If you (or a portfolio company) is confronting these sorts of challenges, maybe I can help? Just reply, or find a time.

In case you missed it…

• 1 hour session on M&A for founders

• Great 30 min session on getting the first 10 customers for earlier stage founders

• Our GTM Assessment checklist, a good framework for anyone (founder or VC) to structure an inquiry into whether GTM is working or not, and how to improve it