AI Agents in Business Software: Where They Actually Earn Their Keep

What an AI agent actually is

A chatbot answers a question. An agent is given a goal, a set of tools it is allowed to call, and permission to decide the order it uses them in. That third part is the whole difference — and the whole risk.

In business software this usually means a model that can read an incoming ticket, look up the related order in your database, check a policy document, and then draft a reply or update a record. Three capabilities separate it from last generation’s chatbot:

• Tool access — it can query your systems rather than guess from training data.
• Multi-step work — it can take the result of one step and decide what to do next.
• State — it remembers what it has already tried inside a single task.

The distinction matters commercially, because the three tiers fail in completely different ways and need completely different amounts of engineering around them.

Most businesses that describe an “agent project” actually need the middle row, and would get 80% of the value for 30% of the work. It is worth being honest about which you are building before the budget is set.

The four workflows where agents pay off first

Across the projects we have shipped, the wins cluster in the same places. They share a pattern: high volume, structured inputs, and a mistake that is cheap to spot.

• Support triage — classifying, tagging and routing incoming tickets, with a drafted first reply that a human approves. Volume is high, the taxonomy is stable, and a misrouted ticket is visible within minutes.
• Quote and proposal drafting — assembling a first draft from a price list, past projects and the enquiry itself. The salesperson still owns the number; they just stop starting from a blank page.
• Data movement between systems — matching records across your CRM, accounting tool and spreadsheets, then flagging what two systems disagree about. Agents are good at the tedious comparison and bad at deciding who is right, which is exactly the split you want.
• Internal knowledge lookup — answering staff questions from your own documents, with the source linked so the answer can be checked. Nothing leaves the building, and the cost of a wrong answer is a follow-up question.

The shared property is verifiability. In each case a human can tell in seconds whether the output is right, which means errors get caught before they compound into anything expensive.

Where agents still fail

Being clear about failure modes is what separates a useful pilot from an expensive one. Four categories account for almost every disappointing result we have seen.

• Irreversible actions — refunds, deletions, outbound payments, messages to customers. The cost of a rare mistake swamps the average saving.
• Ambiguous policy — if two experienced people on your team would answer a case differently, an agent will answer it inconsistently, and you will have no way to explain why.
• Long chains without checkpoints — error compounds. Ten sequential steps at 95% reliability each gives you roughly a 60% chance of a clean run; the same ten steps at 99% gives you 90%.
• Anything with legal or regulatory accountability — a person has to own the decision, so a person has to make it. An audit trail showing “the model decided” is not a defence.

That compounding arithmetic is the most useful thing to internalise. It means the right design is not one agent doing twelve things, but a short agent run inside a workflow that checks its work — or a human who does.

The architecture that keeps them safe

The engineering around the model matters far more than which model you pick. Four things belong in every build:

1. Give the narrowest possible tools. Not “database access” — a function that fetches one order by ID, and another that adds a note to one ticket.
2. Log every tool call with its inputs and outputs, so any outcome can be reconstructed weeks later when somebody asks what happened.
3. Require explicit human confirmation on anything that writes, sends or spends, and make the confirmation show exactly what is about to happen.
4. Build a fixed test set of real past cases and re-run it on every prompt or model change.

Evaluation is the step everyone skips

Without a test set, every change is a guess. You tweak a prompt, the three cases you happen to try look better, and you ship a regression you will discover through complaints a fortnight later.

It does not need to be sophisticated. Fifty real cases with known good outcomes, run automatically, with the pass rate recorded, is enough to turn prompt engineering from folklore into engineering. Add each new failure to the set as it is found, and the suite gets more valuable the longer the system lives.

Permissions belong to the user, not the agent

An agent acting on behalf of a junior staff member should see exactly what that person can see. Agents given blanket service-account access become the most attractive target in your infrastructure, and the least auditable one.

What it actually costs to run

Model pricing is the number people quote and rarely the number that decides the business case. There are four costs:

• Usage — per request, and higher for agents than chatbots because each task makes several model calls.
• Engineering — the tools, permissions, logging and evaluation harness. In year one this is normally the largest line.
• Review — the human time spent approving or correcting output, which is real cost until the accept rate is high.
• Monitoring and maintenance — models and APIs change, and a system nobody watches degrades quietly.

A workable way to sanity-check the case: estimate hours saved per week on the target workflow, multiply by loaded cost, and compare against a year of all four lines above. If the answer is not obviously positive with conservative assumptions, choose a different workflow rather than arguing about the assumptions.

How to start without betting the business

1. Pick one workflow and measure it for two weeks — volume, handling time, error rate. Without a baseline you cannot prove anything afterwards, and you will be arguing about impressions.
2. Run the agent in draft mode only. It proposes; your team accepts, edits or rejects. Those edits are your training signal and your evaluation set.
3. Review the accept rate weekly. Above roughly 80% on a low-risk workflow, widen the scope; below 50%, the workflow is wrong, not the model.
4. Only then automate the accepted path, keeping a human in place for exceptions and anything the agent flags as uncertain.

At ninety days a healthy pilot looks like this: one workflow, a measured before-and-after, an accept rate you can quote, a test suite that runs on every change, and a clear decision about whether workflow two is worth doing. A pilot that has instead spread across six half-finished use cases with no baseline is the common failure, and it usually ends with the conclusion that “AI did not work for us”.