What Happens to an AI System After It Goes Live?

A forecast trained on last year’s orders knows nothing about the customer you won this spring. Every AI system starts going stale the day it goes live. Each question costs a little to answer, your data and systems change, and the AI companies update their models. Plan for upkeep from day one: regular checks against known answers, someone watching the results, and a clear owner. That keeps the system useful year after year, long after the excitement of launch day has passed.

Running costs are small and steady

Most of the money in an AI project goes into building it, as I explain in why an AI project costs real money. Once it is live, the running costs are smaller. They come in three parts.

• Usage. Each time someone asks the AI a question, the AI company charges a small fee. It is usually tiny per question, so the bill depends on how much your team uses it.
• Hosting. The tools, the data links and the screens your team uses have to run somewhere. That is a server bill, like any other software.
• Upkeep. Someone has to check the answers, fix what breaks, and adjust it as your business changes.

The first two are easy to predict. The third is the one people forget, and it is the one that decides whether the system is still useful in two years.

Your business changes and the AI doesn’t notice

An AI system doesn’t know when what you want has changed. It learns from your data as it was on the day it was built. Then your business moves on. You win a new customer with different habits. You drop a product line. A supplier changes its lead times. Metal prices move. Your team starts recording jobs a slightly different way.

The system keeps answering as if nothing happened. This slow slide is called drift. The answers get a bit less right each month, and they still sound just as sure of themselves. Nobody gets an error message. The numbers just quietly stop matching what your best people would have said.

Any tool built on rules works this way. A price rule is only right while the costs and methods behind it stay the same. When those change, the rule has to change too, and the people who do the work are best placed to spot an answer that looks wrong.

Model updates change answers

There is a second kind of change, and it comes from outside your business. The AI companies keep updating their models. Sometimes they retire old ones completely. OpenAI’s own guidance says it plainly: “Software relying on OpenAI models may need occasional updates to keep working.”

Updates can change answers in ways nobody planned. In 2023, researchers from Stanford and UC Berkeley tested two versions of GPT-4, released three months apart, on the same questions. They found that the “performance and behavior of both GPT-3.5 and GPT-4 can vary greatly over time.” Some answers got better and some got worse, with no change on the user’s side.

You can’t stop the AI companies from changing their models. You can build your system so a change hurts less. When the exact work is done by small, tested tools, and the AI only decides which tool to call, a new model has much less room to change your numbers. I cover this in more detail in why good tools beat a smarter AI.

Good upkeep is boring and regular

Keeping an AI system healthy needs very little drama. It needs three habits.

1. Known-answer checks. Keep a set of questions where you already know the right answer: last quarter’s job costs, a few past quotes, a handful of orders. Run them on a schedule, and again after any change to your systems or the model. If the score drops, you know before your team does.
2. Someone watching the results. Pick a person who uses the system and knows the work. Their job is to say “that looks off” out loud, and to have somewhere to say it.
3. One clear owner. One named person in your business is responsible for the system. They decide when it needs fixing and when it should stop being used.

None of this is clever. It is the same discipline you already apply to a machine on the shop floor: a service schedule, an operator who knows how it should sound, and someone accountable for it. An AI system that nobody owns will drift until someone makes a costly decision on a bad answer.

This is also part of what you are paying for when you hire someone to build AI. A good builder plans the checks and the handover from the start. I go into that in what you really pay an AI expert for.

One fee covers build, hosting and usage

Because the running costs and the upkeep matter so much, I don’t hand over a system and walk away. On the AI Builds page you will see that each project has one fee, agreed before work starts, covering the build, the hosting and the AI usage. We keep it working after it goes live.

That means you don’t have to track usage bills, model updates or a vendor’s retirement notices yourself. You still own the results, and your people still say when something looks wrong. We handle the checks, the fixes and the changes behind the scenes.

Plan the upkeep before launch day

If you would rather run it all in-house, that can work too. Budget for the upkeep from the start, write down your known-answer checks, and name the owner before launch day.

Already have a system nobody checks? Book a call from the AI Builds page, and I’ll tell you what it needs to stay right. If you’re still planning one, bring the plan and we’ll work out the upkeep before launch.