The AI automation examples that actually run in companies today are ticket triage, document extraction, reply drafting, meeting-note capture, lead scoring, alert triage and content repurposing. In every one of them the model does exactly one thing: it makes a single judgment call inside a pipeline that is otherwise ordinary software.
Our position is that the department a use case belongs to tells you almost nothing. Sales, marketing, finance, support, IT: the same list gets resorted into those bins on every page you read this week. What decides whether an automation survives contact with production is the type of judgment you ask the model to make, because that one choice sets the failure mode, the amount of human review needed, and therefore whether the economics work at all.
That is not a theoretical worry. In January 2026 a commenter posting as rtbruhan00 described exactly this on Hacker News: their AI support-triage system looked good in testing, then in production "support agents spent more time correcting AI mistakes than they saved," and costs rose about 30% before the team rolled back to rule-based routing with AI demoted to a suggestion tool (HN item 46731329). The workflow was fine. The judgment type was wrong for the amount of review they could afford.
The short answer: Choose AI automation examples by the judgment the model has to make — classify, extract, draft, rank or decide — because that one step sets the failure mode, the review cost and the controls, while the department it sits in sets none of them.
Last updated: July 30, 2026.
The judgment cut: five deterministic steps around one probabilistic one.
Two disclosures before the catalogue. We have not run a controlled before-and-after test of these workflows in a customer's environment for this article, so every number below is either sourced to a named study or labelled as arithmetic with its inputs stated. And the US Census Bureau's Business Trends and Outlook Survey, the source most writers reach for on business AI adoption, blocked automated access at every tier we tried, so we cite the Federal Reserve's reading of it instead.
What Counts as an AI Automation Example
An AI automation is a workflow where software handles the trigger, the data gathering and the write-back, and a model handles one step in the middle that could not be written as a rule. Sorting a mailbox by sender is automation. Sorting it by what the customer is upset about is AI automation. Vendors also sell this under the names AI workflow automation, intelligent automation and AI business process automation; the labels differ, the anatomy does not. The dividing line is not the tooling, the vendor or the word "agent" on the marketing page; it is whether any step in the chain has an output that cannot be predicted from the input by reading the code.
That line matters because a great many things sold as AI automation examples are rule engines with a language model bolted to the front for the demo, and plenty of genuinely useful automation has no AI in it at all and should stay that way. The best outcome of reading a list like this one is sometimes deciding that a scheduled query and a webhook will do.
Adoption is broad enough that this is a mainstream operational question rather than an experiment. Stanford HAI's 2026 AI Index puts organizational adoption at 88%. But "the organization uses AI" and "a process runs on AI unattended" are different claims, and the gap is wide: the Federal Reserve's April 2026 review of the adoption surveys records firm-level adoption in the Census Bureau's Business Trends and Outlook Survey at "about 18 percent of firms at the end of 2025," worker-level generative AI use at "about 41 percent of the workforce," and an employment-weighted rate of "around 78 percent" in the Survey of Business Uncertainty. Three defensible numbers, three different questions. Big firms have adopted; most firms have not; and most individual processes still have a person in the middle of every run.
What an AI automation example is not. Not a product tier, not a chatbot on a website, not a headcount plan, and not a single monolithic agent that "runs your operations." Nearly every deployment that works in 2026 is narrow: one trigger, one judgment, one write.
The Judgment Cut: The Anatomy Every Example Shares
Take any AI automation example and cut it into six steps. Trigger, gather, judge, check, write, record. Five of those steps are ordinary software you can unit-test. One of them is a model producing an output you can only measure statistically. We call that split the judgment cut, and running it on a proposed workflow takes about ten minutes and changes most conversations about it.
The cut moves the argument off the word "AI" and onto a specific question: what exactly is the model being asked to decide, and what happens to that decision next? "Read the invoice and fill in the fields" and "read the invoice and pay it" share five of six steps. They differ in one, and that difference is the whole project.
| Step | What it is | Who owns it | How you test it |
|---|---|---|---|
| Trigger | An event: a ticket lands, a file arrives, a schedule fires | Software | Deterministic tests |
| Gather | Pull the record, the history, the related documents | Software | Deterministic tests |
| Judge | The model produces one output that could not be a rule | Model | Measured on a labelled sample |
| Check | Thresholds, allow-lists, policy, human approval | Software and people | Deterministic tests |
| Write | The action lands in a system or reaches a person | Software | Deterministic tests, plus a rollback path |
| Record | Who ran it, what it decided, what it cost, what was refused | Platform | Audit query |
Most failed pilots we read about are failures of the check and record steps, not the judgment step. The model was roughly as good as expected; nobody had decided what happens when it is wrong, or built the record that would let anyone find out. That matches McKinsey's 2026 AI Trust Maturity Survey of roughly 500 organizations, run between December 2025 and January 2026: "nearly two-thirds of respondents cite security and risk concerns as the top barrier to fully scaling agentic AI, well ahead of regulatory uncertainty or technical limitations," while only about one-third report maturity levels of three or higher in strategy, governance and agentic AI governance (McKinsey). Capability is not the constraint. Confidence in the wrapper is.
This is a different sorting key from the one we used in our earlier analysis of AI agent use cases ranked by blast radius, and the two are complementary. Blast radius answers "how much damage can this do," which sets the sequence you adopt in. The judgment cut answers "what is the model actually deciding," which sets the controls and the review budget for whichever one you picked.
It also tells you what to buy. "Which of eight queues" needs a labelled sample and a confidence threshold. "What should this email say" needs a reviewer and a tone standard. "Should we refund this" needs a limit, an approval and a reversal procedure. Three different projects, three different budgets, and no department-based list will tell you which one you are starting.
Here is a short, current explainer on why the check step — the human in the loop — keeps turning out to be the hard part, from IBM's technology channel:

Five Judgment Types and Their Failure Modes
There are five judgments a model can make inside a business workflow, and each fails in a characteristic way. Classification fails silently into the wrong bucket. Extraction fails silently into the wrong field. Drafting fails visibly, in front of someone who can catch it. Ranking fails slowly, by drifting. Deciding fails once, expensively, often externally. Knowing which one you are buying tells you what to build around it before you write a line of the prompt.
| Judgment | Output | Characteristic failure | Cheapest effective control | The metric that matters |
|---|---|---|---|---|
| Classify | One label from a known set | Confident misroute; the item looks handled | Confidence threshold with a fallback queue | Misroute rate, and cost per misroute |
| Extract | Structured fields from unstructured input | A plausible value that was never in the document | Field-level validation against the source; totals that must reconcile | Fully-correct-document rate, not field accuracy |
| Draft | Text a human sends or publishes | Wrong tone, invented fact, wrong recipient context | Mandatory human send; never auto-send externally | Edit rate and time-to-send |
| Rank | An ordering or a score | Quiet drift as inputs change; entrenched bias | Periodic re-scoring of a held-out sample | Rank stability and outcome lift versus a control |
| Decide | A committed action | One irreversible external action, taken wrongly | Value limit plus approval gate plus reversal path | Reversal rate and time-to-reverse |
Two of these deserve a warning. Classification failures are underestimated because a misrouted item does not look like an error; it looks like a handled item in the wrong place, surfacing days later as an SLA breach. That is the shape of the 30% cost increase in the Hacker News account above: a misroute rate of 15% to 20% sounds survivable until it forces review of every decision, including the 80% the model got right.
Extraction failures are underestimated most. A commenter posting as lumost put the QA problem on Hacker News more clearly than most vendor documentation does, noting that a system may "hallucinate realistic column values rather than extracting its assigned input," and asking how you would catch that without redoing the task by hand (HN item 45823362). Human spreadsheet errors have recognisable shapes: a shifted column, an out-of-range number. A fabricated but plausible value has no shape at all.
There is also a compounding effect that headline accuracy numbers hide. If a model gets each field right 97% of the time and a document has twelve fields, the share of documents entirely correct is 0.97 to the twelfth power, about 69%. That is arithmetic, not a benchmark, and it is why field accuracy is the wrong metric for a business case. The right one is the proportion of items needing no human touch at all.
Independent measurement supports treating reliability, not capability, as the binding constraint. The AI Index's technical performance chapter reports that on the OSWorld computer-use benchmark "accuracy rose from roughly 12% to 66.3%, within 6 percentage points of human performance," and in the same breath that agents "still fail roughly one in three attempts on structured benchmarks." A step that fails a third of the time is perfectly deployable behind a check and catastrophic without one.
AI Automation Examples at a Glance
Eighteen AI workflow automation examples that companies are actually running, sorted by judgment type rather than by department. Read the failure column first; it is the one that determines what the project costs.
| # | Example | Judgment | Typical trigger | How it fails | Minimum control |
|---|---|---|---|---|---|
| 1 | Support ticket triage and routing | Classify | Ticket created | Confident misroute | Confidence threshold, fallback queue |
| 2 | Inbound enquiry categorisation | Classify | Form or email arrives | Sales lead filed as support | Human spot-check of low-confidence items |
| 3 | Expense and transaction coding | Classify | Card feed or receipt upload | Wrong GL code, found at close | Rules for high-value codes, sampling |
| 4 | Security alert and log triage | Classify | Alert fires | Real incident marked benign | Never suppress; only re-order severity |
| 5 | Invoice and PO line capture | Extract | Document lands in inbox | Plausible invented value | Totals must reconcile before write |
| 6 | Contract clause extraction | Extract | Contract uploaded | Missed clause reads as absent clause | Extraction is a checklist for a human, not an answer |
| 7 | Meeting notes to action items | Extract | Recording finishes | Attributed decision nobody made | Owner confirms before anything is created |
| 8 | Resume parsing into an ATS | Extract | Application submitted | Parsing error becomes a rejection | Never let parsing gate a candidate |
| 9 | First-draft support replies | Draft | Ticket assigned | Confident wrong answer to a customer | Agent must press send |
| 10 | Outbound email personalisation | Draft | Prospect enters sequence | Wrong company facts, wrong tone | Human review; no auto-send |
| 11 | Internal research briefings | Draft | Weekly schedule | Fabricated citation | Links must resolve; brief is input, not output |
| 12 | Content repurposing | Draft | Long-form asset published | Off-message claim shipped publicly | Editorial approval before publish |
| 13 | Lead scoring | Rank | New or updated record | Score drifts; reps stop trusting it | Quarterly re-scoring against outcomes |
| 14 | Backlog and bug prioritisation | Rank | Issue created or updated | Loud reporters outrank real severity | Ranking is advisory; owner reorders |
| 15 | Candidate shortlisting | Rank | Application pool closes | Legally high-risk automated evaluation | Treat as EU AI Act high-risk; human decides |
| 16 | Auto-refunds under a threshold | Decide | Refund request | Money leaves for the wrong reason | Hard value cap, reversal path, daily total limit |
| 17 | Autonomous outbound sending | Decide | Sequence step due | Damage to a relationship you cannot recall | Do not; keep a human on send |
| 18 | Auto-close or auto-resolve tickets | Decide | No reply for N days | Unresolved problem marked resolved | Reopen path, and count reopens as failures |
Classify: Examples That Sort Things Into Buckets
Classification is the cheapest judgment to buy and the easiest to get wrong quietly. The model reads an item and returns one label from a set you defined. Because the output space is small and known, a few hundred labelled examples measure it properly, and a confidence threshold sends everything below the line to a person untouched. Almost every classification failure we have read about comes from skipping that threshold.
1. Customer support ticket triage and routing. The judgment: which queue, which priority, which product area. Where it breaks: the model is confident and wrong, the ticket sits in the wrong team's backlog, and the cost surfaces as a missed SLA rather than as an error. What makes it shippable: route only above a confidence threshold, send everything else to the existing manual queue, and publish the misroute rate weekly. The failure recounted on Hacker News is the canonical version of getting this wrong. Review of every decision destroyed the saving, because the review policy was "check all" rather than "check the uncertain ones." That is the difference between the version that works and the version that gets rolled back, and it is a change to the review policy, not to the model.
One caveat that rarely gets said: a language model's self-reported confidence is not calibrated out of the box, so do not assume 0.85 means anything until you have plotted accuracy against confidence on your own sample. Where the model gives you nothing usable, cheap substitutes work — agreement between two runs at different temperatures, or a second small model asked only "does this label fit this text," with disagreement routed to a person.
2. Inbound enquiry categorisation. The judgment: is this a sales enquiry, a support request, a partnership pitch or noise. Where it breaks: a high-intent buyer gets filed as support and answered in two days. What makes it shippable: asymmetric handling. Misfiling support as sales costs a minute; misfiling sales as support costs a deal, so bias the threshold toward the expensive direction rather than toward overall accuracy. It is the commonest tuning mistake we see described: a symmetric accuracy number optimised for an asymmetric business problem.
3. Expense and transaction coding. The judgment: which general-ledger code, which cost centre, which policy category. Where it breaks: errors accumulate invisibly and appear at period close, when the person who could explain the transaction has forgotten it. What makes it shippable: keep rules for the codes that matter, anything tax-relevant or capitalised, let the model take the long tail, and sample a fixed percentage monthly rather than reviewing everything.
4. Security alert and log triage. The judgment: is this alert worth a human's attention now. Where it breaks: a real incident is classified as benign and never seen again. What makes it shippable: a strict rule that the classifier may re-order and group but never suppress. If the model's output can make an alert invisible, you have built a decide-type automation and should treat it as one.
Extract: Examples That Turn Documents Into Fields
Extraction converts unstructured input into structured fields, and it covers most of what people mean by document and contract review automation. It is also where the biggest apparent wins hide the nastiest measurement problem: the model returns something that looks exactly like a correct answer whether or not it is one. You cannot sanity-check it by glancing at it. You have to compare it against the source document, which is most of the work you were trying to avoid.
One design rule follows. Make the deterministic half check the probabilistic half. Totals must reconcile. Dates must parse. Supplier names must match the vendor master. Every field you can validate against something outside the model is a field you no longer have to read.
5. Invoice and purchase-order line capture. The judgment: which strings on this page are the invoice number, the dates, the tax and the line items. Where it breaks: a fabricated but plausible value, or a line-item table read one row out of alignment. What makes it shippable: refuse to write anything that does not reconcile. Line items sum to the subtotal, subtotal plus tax equals the total, the supplier exists in the master, and failures go to a human rather than a guess. Header fields are far more reliable than line-item tables in every evaluation we have seen; treat those as separate automations with separate targets.
6. Contract clause extraction. The judgment: in a document and contract review workflow, which paragraphs are the liability cap, the termination rights, the auto-renewal, the data-processing terms. Where it breaks: a missed clause reads as an absent clause, and silence is indistinguishable from a negative finding. What makes it shippable: frame the output as a checklist pointing a reviewer at page numbers, never as a conclusion. "Clause not found" must render as "not found, check pages 4 to 9," not as "no auto-renewal."
7. Meeting notes into action items and record updates. The judgment: what was decided, who owns it, what changed about the deal or the project. Where it breaks: an action item attributed to someone who never agreed to it, or an open question recorded as a decision. What makes it shippable: the named owner confirms before anything is created. A draft task list in a channel is useful; auto-created tasks with due dates become a trust problem within two weeks.
8. Resume parsing into an applicant tracking system. The judgment: which text is the employer, the dates, the qualifications. Where it breaks: an unusual layout parses badly and the candidate silently drops below a filter. What makes it shippable: parsing may populate fields for a human to read; it may not gate anyone. Under the EU AI Act, systems used "for the recruitment or selection of natural persons, in particular to place targeted job advertisements, to analyse and filter job applications, and to evaluate candidates" are high-risk under Annex III. The moment extraction feeds a filter, the compliance posture of the workflow changes.
Draft: Examples That Write Something a Human Sends
Drafting is the friendliest judgment type, for one structural reason: the failure lands in front of a person before it reaches the world. A hallucinated fact in a draft reply is an editing task. The same hallucination in an auto-sent reply is a liability. That difference explains why drafting automations succeed more often than the rest of the catalogue, and why removing the human is the most expensive instinct in this field.
The best-measured example we have is a drafting one. In the Quarterly Journal of Economics, Brynjolfsson, Li and Raymond studied a generative AI assistant rolled out to 5,172 customer-support agents and found productivity, measured as issues resolved per hour, rose 15% on average. The distribution is the interesting part: "less experienced and lower-skilled workers improve both the speed and quality of their output, while the most experienced and highest-skilled workers see small gains in speed and small declines in quality." Read as a deployment guide, that says give the drafting automation to the newest team and expect little from it on your best people.
9. First-draft support replies. The judgment: what this customer is asking and what a good answer looks like. Where it breaks: a confident wrong answer an inexperienced agent approves because it reads well. What makes it shippable: the agent presses send, the draft cites the knowledge-base article it drew from, and edit rate is tracked per agent. Edit rate collapsing toward zero is a warning about review quality, not a success signal.
10. Outbound email personalisation. The judgment: what to say about this company or person that makes the message worth reading. Where it breaks: stale or wrong company facts, or a tone that reads as generated to precisely the buyers you want. What makes it shippable: a human on send, and a rule that every personalised claim traces to a field in the record rather than the model's memory. Even aggressive vendor content tends to concede this one, and it is right to.
11. Internal research and competitive briefings. The judgment: what matters in this pile of material and how to summarise it. Where it breaks: a fabricated citation, or a confident summary of a source that says something else. What makes it shippable: every link must resolve and be opened by the person acting on it. The brief is an input to a human decision, never the decision.
12. Content repurposing. The judgment: which parts of a long asset become a post, a newsletter, a summary. Where it breaks: an off-message claim ships publicly with the company's name on it. What makes it shippable: the same editorial approval any human draft would get. Volume is not the constraint in content operations; approval capacity is, and multiplying drafts without expanding approval just moves the queue.
Rank: Examples That Decide What Gets Attention First
Ranking is the quietest of the five. Nothing visibly breaks; the order just becomes less useful as inputs shift, and the people consuming it stop trusting it without filing a complaint. Ranking automations need a scheduled evaluation more than a threshold, because there is no per-item error to catch.
13. Lead scoring and pipeline prioritisation. The judgment: which records deserve attention this week. Where it breaks: drift. The score was fitted to a market, a product mix and a source blend that have all moved. What makes it shippable: re-score a held-out sample against actual outcomes on a fixed schedule, and publish the lift over a simple baseline such as recency or company size. If the lift is not measurable against that baseline, the ranking is decoration.
14. Backlog and bug prioritisation. The judgment: what should be worked next, given severity, frequency and customer impact. Where it breaks: the loudest reporters and the most verbose tickets outrank quieter, more serious problems. What makes it shippable: keep the ranking advisory, let the owner reorder freely, then track how often they do. A high override rate is not a failure; it is the measurement you wanted.
15. Candidate shortlisting. The judgment: which applicants go to the hiring manager. Where it breaks: legally, before it breaks technically. What makes it shippable: in the EU, treat it as high-risk under Annex III point 4(a), including the human oversight obligations in Article 14. Our own position is more conservative than the regulation: ranking humans against each other is the one place here where model quality is not the deciding factor, and we would keep it a search aid rather than a shortlist.
Decide: Examples That Act Without Asking
The decide tier is where an AI automation commits an action with no person between the model and the consequence. It is also where most of the value people imagine when they read a list like this one, which is why it is worth being blunt: almost nothing here should be built first, and some of it should not be built at all.
OWASP names the underlying vulnerability directly. In the 2025 LLM Top 10, Excessive Agency is "the vulnerability that enables damaging actions to be performed in response to unexpected, ambiguous or manipulated outputs from an LLM." Its root causes are excessive functionality, excessive permissions and excessive autonomy, and the recommended controls are least privilege and requiring "a human to approve high-impact actions before they are taken." That is a security framing of the design rule the economics already implies.
The case law is ahead of most internal policies. In Moffatt v. Air Canada, 2024 BCCRT 149, British Columbia's Civil Resolution Tribunal rejected the argument that a chatbot was answerable for its own statements, holding that "it should be obvious to Air Canada that it is responsible for all the information on its website," and awarded $650.88 for negligent misrepresentation (decision text). The amount is trivial. The principle is not: an automated statement to a customer is your statement.
The fair counterargument is that the decide tier is where the headcount savings actually live, and that a blanket "do not" is a counsel of despair from people who will not have to hit next year's cost target. That is true, and the position we hold is narrower than a ban. The decide tier becomes reasonable when three conditions hold together: the action is reversible by an ordinary employee within minutes, there is a hard value or volume cap that fails closed, and you have run the workflow in draft mode long enough to have a measured reversal rate. Meet all three and automate. Meet two and keep the approval. What we object to is not autonomy; it is autonomy bought before the reversal path exists.
16. Auto-refunds and auto-approvals below a threshold. The judgment: does this request meet the policy. Where it breaks: money leaves for a reason nobody can reconstruct, or small approvals add up. What makes it shippable: a hard per-item cap, a daily aggregate cap, a reversal path a support agent can execute without engineering, and a log that records refusals as well as approvals. Missing any of the four, keep a human on the approval.
17. Autonomous outbound sending. The judgment: what to say to a prospect and when to say it, with nobody reading it first. Where it breaks: relationships and domain reputation, neither of which has an undo. What makes it shippable: in our view, nothing does at current reliability. This is the clearest "do not" in the catalogue. Keep the drafting; keep the human on send.
18. Auto-close and auto-resolve. The judgment: is this ticket finished. Where it breaks: an unresolved problem is marked resolved and the customer's next contact starts from zero. What makes it shippable: a one-click reopen path, and a metric that counts reopens as automation failures rather than new tickets. Without that accounting change, auto-close always looks like a success.
Where the Money Actually Goes
The model call is almost never the cost. In Anthropic's published pricing documentation, Claude Haiku 4.5 is listed at $1 per million input tokens and $5 per million output tokens, and the same page works a support example: about 3,700 tokens per conversation, roughly $37 per 10,000 tickets. That is $0.0037 per ticket for the judgment step, fetched on 30 July 2026.
Now price the review. At a fully loaded $40 an hour, a stated assumption rather than a survey finding, one minute of human checking costs about $0.67 — roughly 180 times the model call. Every serious cost question about an AI automation is a question about review minutes, not tokens.
That gives the second half of the judgment cut, a rule we apply whenever someone brings us a workflow. Call it the review ratio:
An AI automation pays when the time to verify an output, plus the expected time to fix the ones that are wrong, is less than the time to do the task by hand.
Written out: V + (e x R) < D, where V is minutes to verify, e is the fraction needing rework, R is minutes to rework, and D is minutes to do it manually. Three terms get estimated and one gets ignored — V, the cost of checking, which nobody measures before the pilot because in the pilot the builder checks everything for free.
| Workflow | D: do it manually | V: verify the output | e: needs rework | R: rework | V + eR | Pays? |
|---|---|---|---|---|---|---|
| Route a ticket | 0.5 min | 0.2 min | 0.15 | 3 min | 0.65 min | No, if every item is checked |
| Route a ticket, threshold at 0.85 | 0.5 min | 0.2 min on the 20% below threshold | 0.03 | 3 min | 0.13 min | Yes |
| Extract a 12-field invoice | 4 min | 1.5 min | 0.30 | 4 min | 2.7 min | Yes, narrowly |
| Extract, with reconciliation checks | 4 min | 0.3 min on the 25% that fail checks | 0.05 | 4 min | 0.28 min | Yes, clearly |
| Draft a support reply | 6 min | 2 min | 0.25 | 4 min | 3.0 min | Yes |
Those numbers are illustrative and stated as such; the point is the shape, not the digits. Two things fall out immediately. A "check everything" review policy usually destroys the case on its own, which is the mechanism behind the 30% cost increase in the Hacker News account. And the engineering that matters most is almost never prompt tuning; it is the deterministic checks that shrink V by telling you which outputs are worth looking at.
There is a measurement trap here too. METR's randomised trial of experienced open-source developers found that "when developers are allowed to use AI tools, they take 19% longer to complete issues," while the same developers estimated afterwards that AI had made them about 20% faster (METR, July 2025). We checked whether that still stands. METR's February 2026 update says the team now believes "it is likely that developers are more sped up from AI tools now," but that selection effects make the newer data "only very weak evidence" for the size of the change, with 30% to 50% of developers declining to submit tasks they did not want to do without AI. The safe conclusion is not "AI slows people down." It is that self-reported time savings are unreliable in both directions, so the review ratio has to be measured with a stopwatch on a sample rather than collected from a survey.
Model spend does eventually matter at volume, and it is worth routing cheap judgments to cheap models — we covered how that arithmetic works in our earlier analysis of AI model routing and LLM costs. But if the automation is losing money, the tokens are not why.
Why the Same Example Works Here and Fails There
The same automation example succeeds at one company and gets rolled back at another, and the difference is almost never the model. It is input variance, volume and exception rate. A support queue with clean, templated inbound and eight well-separated categories is a different problem from one where half the tickets arrive as a forwarded email chain with an attachment. Same workflow diagram, different judgment step.
That Hacker News account names its own root cause precisely, and it is the most transferable lesson here: "our training data was too clean compared to real customer queries." That is a sampling problem, not an AI problem. Any evaluation built from tidy examples overstates production accuracy, and the overstatement is largest exactly where the automation was supposed to help most: the messy tail.
| Condition | Automation tends to work | Automation tends to fail |
|---|---|---|
| Input variance | Templated, structured, consistent source systems | Free-form, forwarded chains, mixed languages, scans |
| Volume | Thousands of items per month | Dozens, where a person already has full context |
| Category separation | Distinct, mutually exclusive labels | Overlapping labels that humans also argue about |
| Exception rate | Under roughly 10%, and exceptions look different | High, and exceptions look like normal items |
| Reversibility | The write can be undone in minutes | The action is external or financial |
| Ownership | One named owner who feels the errors | Shared ownership, nobody watches the queue |
Volume cuts against the instinct to start small. A workflow with thirty items a month cannot generate enough signal to tell whether the automation is working, and the person handling those thirty items already carries the context the model is missing. Low-volume, high-context work is the worst possible first project, and it is very often the one chosen because it annoys somebody senior.
A broader pattern shows up in the failure statistics. Widely cited 2025 research put the share of enterprise generative AI pilots producing no measurable profit-and-loss impact at around 95%; we could not retrieve that report from a primary source at any fetch tier, so we mention it only as the number readers will have seen and do not treat it as established. What we can cite is the direction: McKinsey's trust survey finds active mitigation lagging risk awareness across nearly every category, around 8% of organizations reporting AI-related incidents, and almost 60% of those who had one rating their organization's response as merely satisfactory or worse. Pilots do not usually fail because the model was bad. They fail because nobody built the part that catches the model being bad.
The Examples That Should Not Be AI at All
Some of the most satisfying automation wins have no model in them. If the input is structured, the rule is stable and the output is checkable, a scheduled query and a webhook beat an AI automation on cost, latency, auditability and everything else that matters at 3am. An honest catalogue has to include the cases where the answer is "not this."
Four tests, and if any of them passes, write the rule instead:
- The input is already structured. Routing by product SKU, region or contract tier is a lookup. Adding a model adds variance to a step that had none.
- The rule is stable and written down somewhere. If the policy exists as a document a person follows exactly, encode the document. A model is for the cases the document does not cover.
- The output is fully checkable by a rule. If you can validate the answer deterministically, you can usually produce it deterministically.
- The consequence is irreversible and external. Wire transfers, contract signature, public statements, anything that touches a person's employment or credit. Reversibility is the property that buys you the right to be wrong occasionally.
Test four has a regulatory version, worth reading rather than summarising. Annex III of the EU AI Act designates as high-risk, among others, systems used to evaluate creditworthiness, to price life and health insurance, to triage emergency calls, and to make "decisions affecting terms of work-related relationships, the promotion or termination of work-related contractual relationships." Several items that appear casually on automation listicles sit inside that list. Our earlier guide to EU AI Act compliance for AI deployers covers what the deployer obligations require.
A final case for the rules: the hybrid. The strongest deployments we have looked at are not AI-versus-rules but rules with a model in the gap. The rule engine handles the items it can match with certainty, the model handles the ones the rules miss, and a person handles what is left. That gives you a deterministic majority, a measurable middle, an explicit exception path, and graceful degradation when the model is unavailable, which a pure AI workflow does not have.
Run the Judgment Cut in Two Weeks
This is the part most catalogues leave out: how to get from a list of examples to a decision about one of them, without a platform purchase.
Before you start, you need four things. A workflow with at least 200 items a month. One named owner who currently feels the errors. Read access to 100 real historical items, including the ugly ones. And a decision, written down, about what happens to an item the automation gets wrong.
If you cannot supply the fourth, stop. Everything downstream depends on it.
The exercise is not free. Our estimate is three to five person-days across the two weeks, most of it the manual labelling in days 3 and 4, plus a few hours of the owner's time. That is the price of finding out, and it is far below the price of discovering the same thing after a team has reorganised around the workflow.
Days 1 to 2. Cut the workflow. Write the six steps out: trigger, gather, judge, check, write, record. Name the judgment in one sentence that starts with a verb and names the output type. "Classify the ticket into one of eight queues." If the sentence needs an "and," you have two judgments and should pick one.
Days 3 to 4. Build the sample, ugly first. Label 100 real items by hand and deliberately over-sample the difficult ones: forwarded chains, attachments, multi-issue items, non-English. A random sample from last month will be too clean, the documented root cause of the rollback described earlier. Record how long labelling takes per item. That number is D.
Days 5 to 7. Measure the judgment step alone. Run the model over the sample with no automation attached. Record accuracy overall, accuracy by segment, and the confidence distribution. Then measure V by timing yourself verifying 20 outputs against the source. Most teams find here that V is much larger than assumed, and that most of it goes on items the model got right.
Days 8 to 9. Find the threshold, not the prompt. Sort the sample by confidence and find the point above which errors become rare. That threshold and its fallback queue are worth more than any prompt engineering you will do. Then design the deterministic checks that shrink V further: reconciliation, cross-references, allow-lists.
Days 10 to 12. Shadow-run. Let the automation produce outputs nobody acts on, alongside the human process, for a few hundred real items. Compare. This is the only step that produces a number you can defend in a business case, and the one most often skipped.
Days 13 to 14. Decide with the review ratio. Put your measured V, e, R and D into V + (e x R) < D. If it fails, do not tune the prompt; change the review policy, the threshold or the scope, then re-run. If it fails again, the workflow is not a candidate. That is a successful outcome for a two-week exercise.
A worked judgment cut, filled in
Here is what the finished artifact looks like for example 1, filled in end to end, so the output of the exercise is concrete rather than a template.
| Field | Entry |
|---|---|
| Workflow | Inbound support ticket triage, 3,400 tickets per month |
| Trigger | Ticket created in the helpdesk |
| Gather | Ticket body, subject, customer tier, product, last 3 tickets |
| Judgment (one sentence) | Classify the ticket into one of eight queues |
| Judgment type | Classify |
| Check | Route only at confidence >= 0.85; below that, existing manual queue |
| Write | Set queue and priority fields; no customer-visible action |
| Record | Ticket id, model, confidence, chosen queue, whether a human overrode it |
| Failure mode | Confident misroute, surfacing as an SLA breach two days later |
| Reversal | Any agent can re-route in one click; overrides are logged |
| D (manual) | 0.5 min per ticket |
| V (verify) | 0.2 min, applied to the ~20% below threshold |
| e / R | 0.03 above threshold / 3 min to fix a misroute |
| Review ratio | 0.13 min < 0.5 min — passes |
| Owner | Support operations lead |
| Kill criterion | Misroute rate above 8% for two consecutive weeks |
The kill criterion is the row people leave blank, and it is what separates a pilot from an experiment. An automation without a written condition for switching it off will not be switched off. It will be defended.
What Production Adds That the Pilot Did Not Have
A pilot has a builder watching every run. Production does not, and six things that were free in the pilot become real work. These are the questions security review will ask, and they are much cheaper to answer before the workflow exists than after.
Identity. The automation acts as something. If it runs on the builder's personal API key or a shared service account, the audit trail says a person did it, offboarding does not remove its access, and nobody can answer "who owns this." Every automation needs its own non-human identity with an owner, a scope and an expiry, the argument we made in our analysis of non-human identity for AI agents.
Scope. OWASP's least-privilege guidance is worth copying literally: an agent reading a product database "might only need read access to a 'products' table; it should not have access to other tables, nor the ability to insert, update or delete records." Scope the connector to the actions the judgment cut requires. A classification automation needs write access to nothing but a queue field.
Approval. For anything in the decide tier, the approval gate is a product requirement, not a policy document. It has to be in the execution path, it has to record who approved and when, and it has to be usable enough that people do not route around it.
Untrusted input. Almost every example here reads text someone outside your company wrote: tickets, emails, invoices, contracts, resumes, web pages. That makes the gather step an injection surface, because instructions hidden in the content can be read by the model as instructions. The control is architectural: treat the model's output strictly as data, put every privileged action behind the deterministic check step, and never let the judgment step decide its own permissions. OWASP says the same thing in security language — authorization belongs in the downstream system, not in the model's reasoning.
Failure and retries. Production adds partial failure, which pilots never see. The write times out after the model call has been paid for; a retry re-runs the chain and creates the record twice. Make writes idempotent with a key derived from the trigger, make runs resumable from the last completed step, and decide whether a retry re-runs the judgment or reuses the stored output. Reusing it is usually right and always cheaper.
Audit. The record step gets cut for time and is the only reason you can answer questions later. It should capture what was decided, what was refused, what data was touched and what it cost. Refusals especially: "the automation declined 40 requests this week" is the evidence that the controls work. We set out what a defensible action record contains in our piece on observability and audit trails.
For sensitive-data workflows, the mechanism most people are groping toward when they ask "can I trust AI with customer data" is redaction before the model call plus a policy check in the request path, which we covered in inline data-loss prevention at the gateway.
Where LeapForce Fits
LeapForce does not build these eighteen automations for you, and we do not sell a support agent, an invoice reader or a sales-outreach tool. We build the layer underneath them: one controlled path for every AI tool, connector, model and agent, so the check, write and record steps of the judgment cut are the same for the twentieth automation as for the first. Our AI Gateway puts every model call through one place that identifies the caller, evaluates policy before egress, routes, executes and attributes cost, and its rollout model is deliberately staged — observe first, enforce second, optimize third — so a team sees what is actually running before anything is blocked. Workflows is where approval gates, budget caps and durable runs live for the decide-tier examples that need them.
In line with the build-status convention we publish on the site: gateway endpoints, tracing and SSO are live today; vaulted credentials, inline data-loss prevention and dollar budgets are in development; shadow-AI discovery and compliance-evidence packs are on the roadmap. Automating one workflow, you need none of this. You need it at automation number five, when five teams have five keys and nobody can answer what any of them touched.
Limits of This Analysis
Everything above is a synthesis of published research, practitioner accounts and vendor documentation, not a controlled study. Four limits are worth stating plainly.
We did not run these workflows. Nobody on our side ran a before-and-after test of eighteen automations for this article. The review-ratio table is arithmetic with declared inputs, and the $40 hourly figure is an assumption you should replace with your own loaded rate. The only measured numbers here belong to the studies they are attributed to.
The judgment types are a lens, not a taxonomy anyone else uses. Real workflows blur. A support reply draft that also sets a ticket category is doing two judgments, and a good extraction system does classification internally. The cut is useful because it forces a decision about which judgment carries the risk, not because the boundaries are crisp.
Accuracy figures move fast and rarely transfer. Published accuracy on a benchmark corpus tells you very little about accuracy on your documents, and any specific number here has a short half-life. The compounding arithmetic will still hold when the base rates change; the base rates will not.
Two sources a reader would expect are missing. The Census Bureau's Business Trends and Outlook Survey and CanLII's copy of the Air Canada decision both blocked automated retrieval at every tier we tried, so we cite the Federal Reserve's reading of the former and a hosted copy of the decision text for the latter. The circulated "95% of pilots fail" figure is excluded from our evidence for the same reason.
Frequently asked questions
AI automation is a workflow in which software handles the trigger, the data gathering and the write-back, while a model makes one judgment in the middle that could not be expressed as a rule. Rule-based automation is fully deterministic: every output can be predicted by reading the code. The practical difference is testing. You unit-test a rule; you can only measure a model on a labelled sample, so an AI automation needs an accuracy target, a threshold and a fallback path that a rule-based one does not.
Start with a classify or draft workflow that runs at least a few hundred times a month, writes to one internal system, and can be undone in minutes by its owner. Customer support ticket triage with a confidence threshold, and first-draft replies with a human on send, are the two most consistently successful starting points. Avoid anything that sends to a customer without review, moves money, or evaluates a person, however impressive the demo.
To assemble one, increasingly not: trigger, gather and write are configuration in most workflow tools. To ship one safely, you need someone who can build the deterministic checks around the judgment step — reconciliation logic, confidence thresholds, validation against a system of record. That work decides whether the automation pays, and it is engineering whether or not it involves writing code.
The model call is usually the smallest line. Anthropic's pricing documentation works an example at roughly $37 per 10,000 support conversations on Claude Haiku 4.5, about $0.0037 per item, fetched 30 July 2026. Human review dominates: at a loaded $40 an hour, one minute of checking costs about $0.67, roughly 180 times the model call. Budget for review minutes and integration work, not tokens.
The evaluation described in this article takes two weeks and needs no new software. Getting the first workflow into production behind a threshold typically takes weeks rather than months when it writes to one internal system, and considerably longer when identity, connector scoping and approval gates have to be built from scratch, because that is a platform project rather than a workflow project. Doing the platform work once is what makes automations two through ten fast.
Ask for mechanisms, not assurances. The ones that matter: redaction or masking of personal data before the model call, a policy check in the request path rather than in a document, credentials brokered so the raw key never reaches the workflow, scoping to the minimum actions needed, retention limits on prompts and outputs, and an audit record of what was touched. If a vendor answers with a paragraph about how seriously they take security instead of naming which six they implement, that is the answer.
Shadow-run it. Let the automation produce outputs nobody acts on, in parallel with the existing process, for a few hundred real items, then compare against what the humans did. Nothing reaches a customer or a system of record, so the risk is close to zero, and the comparison gives you real accuracy and real verification time instead of estimates. Write the kill criterion before you start.
There is no universal threshold; the requirement comes from the review ratio and the reversibility of the write. A classifier at 85% accuracy is fine if the 15% is caught by a confidence threshold and re-routing costs a click. The same 85% is unacceptable for anything irreversible. Watch compounding: a model right on each field 97% of the time returns a fully correct twelve-field document only about 69% of the time, so measure the share of items needing no human touch, not field-level accuracy.
No, and the framing is misleading. Those tools are excellent at five of the six steps in the judgment cut, and most AI workflow automation is built by adding a model call and its integration glue inside one of them. Replace them only when you need things they do not provide: durable long-running executions, per-workflow identity and scoping, approval gates in the execution path, and an audit record of what was refused. Those are governance requirements that arrive at your fifth automation, not your first.
In the EU, Annex III of the AI Act designates recruitment and candidate evaluation, employment decisions such as promotion or termination and task allocation, creditworthiness assessment, life and health insurance pricing, eligibility for essential public benefits, and emergency-call triage as high-risk. Several of those appear casually on general automation lists. If your workflow is in that set, the human oversight requirements of Article 14 apply and the project is a compliance project with an automation inside it.
Four numbers, before and after: the share of items needing no human touch, verification time per item, the rework rate with its cost, and one business outcome the workflow was meant to move, such as time to first response. Do not rely on how much faster the team feels. METR found developers estimating a 20% speedup on tasks a randomised trial measured as 19% slower, so replace perceived savings with a stopwatch on a sample.
They can, and this is where governance stops being optional. Chaining means one workflow's output becomes another's input, so an error in a classification step propagates into a drafting step with no human between them, and the audit question becomes which automation caused an outcome. Chain only when each link has its own identity, scope and record, and put an approval gate wherever the chain crosses from internal writes to external actions.
Ready to Govern Your AI?
Talk to LeapForce — one controlled layer for every AI tool, connector, model, and agent.
Comments