AI · Agents
Your first AI agent: how a mid-size company picks the use case and runs the pilot
Most first AI projects fail at the choice, not the model: the job is too vague, too rare or impossible to measure. Here is how to pick a first use case with volume, clear rules and a number you can check, and how to run an eight-week pilot that ends in a decision.

Start with a job, not a strategy
An AI agent, in practical terms, is software that reads an input (an email, a form, a ticket, a PDF), decides what to do using your rules and past examples, takes an action in your systems, and hands the case to a person when it isn't sure. That last part matters as much as the rest.
The first agent in a 300-person company should be boring. Not a company-wide assistant, not an "AI strategy", but one job a team does hundreds of times a week and would happily stop doing. A boring first project ships, gets measured and earns the budget for the second one.
Three filters for the first use case
1. Volume. Look for a job with, say, 200 or more cases a week. Below that, the time saved won't cover the setup, and you won't collect enough cases in a few weeks to judge quality.
2. Clear rules. A new hire could learn it in a week from a written guide. If the honest answer to "how do you decide?" is "it depends, ask Maria", it is not your first project.
3. A measurable outcome. Time per case, share handled without a person, error rate, response time. You need today's number so you can compare after.
Add a fourth check: what happens when the agent is wrong. A misrouted email costs a few minutes. A wrong refund or a wrong price quoted to a key account costs money and trust. Start where mistakes are cheap and easy to reverse.
Good first candidates:
- sorting incoming email to the right team, with a draft reply;
- "where is my order" questions;
- reading supplier invoices and entering the fields into the ERP;
- qualifying web-form leads and booking the call;
- answering employees' questions about vacation, expenses and IT access.
Poor first candidates: price negotiations, anything where a mistake can't be undone, and processes nobody has written down.
A practical way to choose: list ten candidates, score each from 1 to 5 on volume, clarity of rules, measurability and cost of a mistake, and pick the top one that also has a manager who wants it.
The data you need
- Past cases with the right outcome. Say, 300 to 500 real examples: the email and the reply that was sent, the invoice and the fields that were entered. Three hundred clean examples beat ten thousand where nobody checked the outcome.
- The rules in writing. The guide new hires get. If there isn't one, record an hour with your best person doing the job and write it up.
- The edge cases. Ask the team for the twenty strange ones they remember. They go straight into your test set.
- Access to systems. Read-only first. Write access comes after the agent has earned it.
- Today's numbers. Weekly volume, time per case, error rate. If nobody tracks them, measure for two weeks before you start.
A pilot plan that ends in a decision
- Weeks 1–2: baseline and success criteria. Collect the data, write the rules, and agree in one written sentence what success means. For example: "The agent fully handles 60% of order-status emails, with fewer errors than the team, and the median first reply drops under ten minutes."
- Weeks 3–4: build and test on history. Run the agent on past cases and compare its answers with what people did. Fix the rules, not just the prompts.
- Weeks 5–6: shadow mode. The agent drafts, people send. Track how often a draft goes out unchanged, edited or thrown away.
- Weeks 7–8: limited live. The agent acts on its own for one category or a share of traffic. Someone reviews a sample every day.
- Decision. Go, extend or stop, judged against the sentence from week one.
The pilot needs an owner on the business side, not only in IT. The real cost of the pilot is that team's review time, so put it in their calendar.
What to measure
- share of cases handled without a person, and share handed over, with the reasons (a clean handover is a good outcome, not a failure);
- accuracy on a weekly sample, checked by a person against the written rules;
- time per case and time to first response;
- in shadow mode, how much people edit the drafts;
- cost per case: model usage, hosting and review time;
- complaints, reopened cases and reversals.
Don't report "number of conversations". Volume isn't value.
What to do this week
- Ask three department heads which job their team does most often and likes least.
- Score the candidates on volume, rules, measurability and cost of a mistake.
- For the winner, pull 300 past cases and whatever written rules exist.
- Write down today's numbers.
- Name the owner and agree on the one-sentence definition of success.
Boson Agents, the AI lab of Boson Group, builds agents for this kind of work, from choosing the first use case to running the pilot. Write to team@boson.group and tell us which job you would start with.




