Thirty years in IT has taught me one thing about hype cycles: the technology changes, the failure pattern doesn’t. I’m watching it happen again with “AI transformation” — same mistakes, new vendor decks.
It goes like this: a few teams spin up GenAI pilots. A chatbot here, a copilot there, a “let’s see what it can do” hackathon project somewhere else. Executives get a slide with impressive demo numbers. Six months later, nothing has shipped to production, nobody can say what the ROI was supposed to be, and the org is quietly starting three more pilots to make up for the fact that the first three went nowhere.
This isn’t a technology problem. It’s an architecture and ownership problem wearing an AI costume.
The Pattern: Tools First, Ownership Never
Most GenAI initiatives start with a tool — an LLM API, a copilot license, a vector database — rather than a problem. Someone in the org gets access to a model, builds something impressive in a sprint, and that becomes “our GenAI strategy.”
The result is predictable:
- Scattered pilots that don’t share infrastructure, data pipelines, or evaluation standards
- No single owner accountable for whether any of it actually moves a business metric
- Duplicate spend on overlapping tools because nobody has visibility across teams
- Demo-to-production graveyards — impressive in a meeting, unusable at scale
None of this is because the technology doesn’t work. It’s because GenAI got treated as something you bolt on, not something you architect for.
The Core Mistake: Feature Thinking, Not Capability Thinking
Here’s the reframe that changes everything: GenAI is not a feature. It’s a capability.
A feature is something you add to an existing product — a button, an endpoint, a UI panel. A capability is something that changes how your data flows, how your systems talk to each other, how decisions get made, and what your operational obligations become.
When you treat GenAI as a feature, you ask: “Which tool should we use?”
When you treat it as a capability, you ask harder, more useful questions:
- What data does this actually need, and is it clean, current, and governed enough to trust?
- Who owns the output when the model is wrong — and how do we know when it’s wrong?
- What existing systems does this need to integrate with, and what breaks if it fails silently?
- What’s our process for updating, evaluating, and retiring this over time?
Companies that skip these questions get demos. Companies that answer them get systems that survive contact with production.
Where the Investment Actually Needs to Go
The industry’s attention has been wildly lopsided: heavy investment in prompt engineering and demo polish, minimal investment in the unglamorous plumbing that makes any of it dependable.
In practice, that plumbing looks like:
- Integration — connecting GenAI outputs to real workflows and systems of record, not a standalone chat window
- Evaluation — a repeatable way to measure quality, drift, and failure modes over time, not a one-time “it looked good in the demo”
- Operational readiness — monitoring, fallback behavior, cost controls, and incident response for a system that is probabilistic, not deterministic
Skipping these isn’t a shortcut. It’s borrowing against your future production incident budget.
The Senior Architecture Lens: Start With Value and Constraints, Not With the Model
A pragmatic architect doesn’t start a GenAI conversation with “what can the model do?” They start with three questions, in this order:
- What’s the value? What business outcome are we actually trying to move, and how will we measure it?
- What are the constraints? Data quality, compliance, latency, cost, team skill level, existing tech debt.
- What already exists? What systems, data pipelines, and processes does this need to plug into — and what’s the cost of that integration versus the cost of the AI component itself?
Only after those three are answered does “which model or tool” become a relevant question — and often, it’s a smaller decision than everyone assumed.
A Simple Decision Framework: GenAI vs. Simpler Alternatives
Not every problem that looks like an AI problem needs GenAI. Part of the job of a pragmatic technical advisor is talking clients out of GenAI as often as into it.
GenAI is likely justified when:
- The task involves unstructured input (language, documents, ambiguous requests) that resists rule-based logic
- Outputs benefit from variability, synthesis, or judgment rather than a single correct answer
- The cost of an occasional imperfect output is acceptable and recoverable
- You have (or can build) a real evaluation loop to catch and correct drift
Simpler automation or process change is likely the better call when:
- The task is deterministic and rule-based — a workflow engine or script solves it more reliably and cheaply
- The “problem” is actually a broken process that GenAI would just paper over
- Data quality is poor enough that GenAI would confidently produce plausible nonsense
- You can’t yet measure whether the output is good — which means you can’t yet manage it
If you can’t articulate why GenAI specifically is the right tool — versus better data, a workflow fix, or a simpler rules engine — that’s usually a sign you’re chasing the trend, not the outcome.
What Success Should Actually Look Like
Model accuracy and clever prompting are inputs, not outcomes. They belong in an engineering retro, not a board update.
Real success looks like:
- A measurable business outcome tied to the initiative before it launched, not invented afterward to justify the spend
- A named owner accountable for the system’s performance in production, including when it fails
- A repeatable evaluation process that catches degradation before customers or employees do
- Integration that holds up under real usage patterns, not just the demo script
- A clear cost model — including the ongoing cost of monitoring, evaluation, and human oversight, not just the API bill
If none of your GenAI initiatives can answer “who owns this, how do we know it’s working, and what did it actually move” — you don’t have an AI strategy. You have a portfolio of expensive experiments.