Most Corporate LLM Projects Fail for Boring Reasons

Every few weeks someone sends me the same story with different logos on it. A company ran an LLM pilot. The demo was gorgeous — a chatbot that answered support tickets, a copilot that drafted contracts, an “agent” that triaged the backlog. Everyone in the room nodded. Budget got approved. And six months later the thing is quietly dead, mentioned only in a retro nobody wants to attend.

The reflexive explanation is always the same: the model wasn’t good enough. That’s almost never true. The models are, frankly, embarrassingly capable now. The projects die for reasons that are far more boring — and far more fixable — than “the AI is dumb.”

Let me walk through the ones I keep seeing, because if you’re about to spend a quarter of your engineering budget on a copilot, the failure modes are more useful to know than the success stories.

Photo by Vitaly Gariev on Unsplash

The demo trap

A demo is a controlled environment. You pick the input. You pick the question. You run it three times and screenshot the good one. Of course it looks magical — you’ve removed every source of friction that exists in production.

Production is the opposite. Production is the user who pastes a 40-page PDF with a table that’s actually an image. Production is the question phrased in a way no human would phrase it, at 2 a.m., with a typo in the one word that mattered. Production is the same query returning a confident, beautifully-formatted, completely wrong answer — and nobody noticing for three weeks because it looked right.

The gap between “it worked in the demo” and “it works for ten thousand strangers” is not a model problem. It’s an engineering problem, and most teams treat the demo as if it were 90% of the work. It’s more like 10%. The boring 90% — the part nobody screenshots — is where the project actually lives or dies.

Photo by Winston Chen on Unsplash

The real failure modes

Here’s where the projects I’ve watched actually fall apart, roughly in order of how often I see it.

Your data is a swamp and you knew it. The single most common killer. The LLM is only as good as what you let it read, and what you let it read is a decade of contradictory wiki pages, a Slack channel where the real answers live, and a “source of truth” document last updated by someone who left in 2022. Retrieval-augmented generation doesn’t fix bad data; it launders it. It takes your mess and gives it a confident voice. Companies discover, painfully, that the AI project was secretly a data-cleanup project all along — and nobody budgeted for the unglamorous months of fixing the swamp first.

Nobody owns it. A pilot is exciting; an owner is accountable. The pilot gets a champion — usually someone senior and enthusiastic — but no team whose job it is to babysit the thing at 3 a.m. when it starts hallucinating refund policies. When the champion moves on, the project has the lifespan of a houseplant in an empty apartment.

Expectations were set by marketing, not by the people building it. Someone promised “it’ll handle 80% of tickets.” The honest engineers knew it could safely handle the 30% that are boring and repetitive, and that the other 50% are exactly the cases where being wrong is expensive. The 80% number becomes the bar. The project “fails” against a target that was never real.

The last 20% is 80% of the work, again. This is the oldest law in software and AI did not repeal it. Getting to a thing that’s right most of the time is fast and thrilling. Getting to a thing that’s right reliably, that fails safely, that says “I don’t know” instead of inventing — that’s slow, expensive, and deeply unsexy. Teams sprint through the thrilling part and run out of energy (and budget, and political cover) before the boring part is done.

None of these are about the model. Swap in a model twice as smart and you still have a swamp, no owner, a fake target, and an unfinished last mile.

The same disease, in open source

You can watch this play out in miniature in the open-source world, where the drama is more public and the stakes are personal rather than corporate.

There’s a real, ongoing fight about AI-generated contributions to open-source projects. A maintainer wakes up to a pull request that looks perfect — clean diff, plausible description, tests that pass. And then they spend two hours reviewing it only to realize it solves a problem nobody had, in a way that subtly breaks an invariant the author never knew existed, because the contributor never actually understood the codebase. The machine wrote it. The human shipped it without comprehension.

It’s the demo trap wearing a hoodie. The output looks like work. It passes the superficial check. And it shifts the real cost — understanding, verification, maintenance — onto someone who didn’t consent to it. Maintainers are right to be furious, and the good ones are responding the only sane way: not by banning the tools, but by raising the bar on accountability. If you submit it, you own it. You understood it. You can defend every line. The tool is allowed; the abdication is not.

That’s the whole lesson, compressed. The problem was never the tool generating text. The problem is humans treating generated text as finished work.

What the survivors actually do

The LLM projects that don’t die are almost disappointingly unglamorous. I’ve never seen one survive on model choice alone. They survive on plumbing.

They scope brutally. Not “answer any customer question” but “draft a first-pass reply to the twelve most common shipping questions, which a human approves before it sends.” Narrow enough that the failure cases are knowable and the wins are real.

They build an eval harness before they build the feature. This is the single biggest predictor I’ve found. The teams that win write down, in advance, a few hundred real examples with known-good answers, and they run every prompt and model change against that set. They can measure whether a change made things better or just different. The teams that lose are flying on vibes and screenshots, and they can’t tell a regression from a Tuesday.

They keep a human in the loop where being wrong is expensive — and remove the human where it’s cheap. The art is knowing which is which. A wrong autocomplete is free. A wrong refund is not. A wrong medical summary is a lawsuit. Mature teams don’t ask “can the AI do this?” They ask “what’s the cost of it doing this wrong, and who catches it?”

They treat it as software, not magic. Versioned prompts. Logging. Monitoring for the slow drift where outputs quietly get worse. Rollbacks. The same discipline you’d apply to any system that touches users — because that’s what it is.

The uncomfortable, useful truth

I use these tools constantly. I’m not a skeptic in the “it’s all hype” sense — the capability is real and it’s changed how I work day to day. I’ll happily let a model write the boring scaffolding, the test fixtures, the first draft of a function I can see in my head. That’s genuine leverage and pretending otherwise is silly.

But leverage is the right word. A lever multiplies force; it doesn’t decide where to push. The teams that get burned are the ones that confused the lever for the hand. They expected the model to supply judgment, ownership, taste, and a clean data foundation — none of which come in the box.

So if you’re staring down an LLM project and you want it to still be alive next year, here’s the unsexy checklist. Fix your data first, even though it’s miserable. Give it a real owner, not a champion. Set a target the engineers actually believe. Build the eval harness on day one. Scope it small enough that you can finish the boring last mile. And decide, explicitly, who is accountable when it’s confidently wrong — because it will be, and the question is only whether a human catches it.

The model isn’t going to fail you. The plumbing is. Go fix the plumbing.