How to Create an AI Agent: Spec First, Prompt Later

To create an AI agent, write a one-page spec before you open any builder: one job in one sentence, the exact records it may read and write, a spending ceiling, a named human owner, and thirty test cases with known-correct answers. Then pick a platform, wire the tools, and run the tests. The build itself is one to three days. The spec is the work.

Our position is narrower than that, and it is the part most guides skip: write the test cases before you write the prompt. If you cannot sit down and produce thirty realistic inputs paired with the answer you would accept from a competent employee, the job is not defined well enough to hand to software, and no platform choice will rescue it. Everything downstream is easy once that list exists and guesswork until it does. Model selection. Tool wiring. Guardrails. Budget.

A startup founder posted exactly this problem to Hacker News in January 2025, and it has not aged. Writing as rkuodys in Ask HN: Open-source AI Agents building platform?, they described wanting "internal agents that have way more permissions" than they would ever expose to customers, running in Slack, on self-hosted infrastructure, with access to full company data. That is not a question about which model is smartest. It is a question about how to give software real authority without losing control of it. The replies came back as a list of tools.

The short answer: Creating an AI agent is a five-part specification exercise (job, authority, tools, tests, owner) followed by a short build. Write thirty test cases with known-correct outputs first; if you cannot, the job is not ready to automate.

Last updated: July 30, 2026.

The Agent Spec Sheet: nine fields, one page, written before you open a builder.

One disclosure before the method. We have not run a controlled build of this exact procedure end to end and timed it, so nothing below is presented as a measured result from our own deployment. The reliability and cost arithmetic in this article is ours and is reproducible from the published figures we cite; the procedure itself is synthesised from public benchmark research, vendor documentation and practitioner reporting, all linked inline.

What you need before you start

You need five things before you touch a builder, and none of them is technical: a job that repeats at least weekly, a person who currently does it, a definition of "done" that two people would agree on, a system of record the agent can read, and someone willing to own the thing after launch. Missing any one of these is the most reliable predictor that the project stalls. You do not need all five to build an AI agent. You need all five to keep one.

Notice what is not on that list. You do not need a data science team, a fine-tuned model, or a vector database. You do not need to have chosen a platform. The most common sequencing error in guides on how to create an AI agent is putting platform selection first, which forces every subsequent decision to be made in that platform's vocabulary.

Here is the honest prerequisite table, with the failure that follows when each is missing.

The third row is the one people skip and the one that decides the outcome. Take an hour, sit with the person who does the job, and write down twenty real cases from last month with the answer that was actually given. If two of you disagree about what the right answer was on more than a handful, you have found a process problem, not an AI problem, and building an agent will encode the ambiguity rather than resolve it.

There is a mirror-image prerequisite that vendor guides never mention: you need to know which of your systems the agent will be allowed to change. Reading is cheap and mostly reversible. Writing is neither. We treat the read/write boundary as the real line between a chatbot and an agent, which we argued in our earlier analysis of where the write-access line falls. Decide it before you build. Retrofitting it means rebuilding the tool layer.

Step 1: Write the job as one sentence with a refusal clause

Write what the agent does in one sentence of under 25 words, then add a second sentence that begins "It refuses to..." The refusal clause is not a safety garnish; it is the boundary that makes the job testable. An agent with no stated refusals has no defined edge, and an undefined edge is where every production failure begins.

A weak job statement looks like this: "Handles customer support tickets." A usable one looks like this:

Job. Reads inbound billing tickets, classifies them into one of six categories, drafts a reply using the current pricing page, and posts the draft to the agent's queue in Zendesk. Refusal. It refuses to send any reply directly to a customer, to issue a refund or credit of any size, to answer anything outside billing, and to act when the customer record is missing or the ticket references a legal dispute.

That second block does four things at once. It bounds the test set. It tells you which tools you do and do not need. It tells you what human approval gates you must build. And it gives the model something concrete to detect, because "this is outside billing" is a judgement a model can make, whereas "be careful" is not.

Gartner's June 2025 assessment of the category is blunt about what happens when this step is skipped: over 40% of agentic AI projects will be canceled by the end of 2027 "due to escalating costs, unclear business value or inadequate risk controls," according to Gartner. Anushree Verma, Senior Director Analyst at Gartner, attributed much of it to projects that are "early stage experiments or proof of concepts that are mostly driven by hype and are often misapplied." Unclear business value is what a missing job sentence looks like on a budget line eighteen months later.

Narrow beats clever, and the reason is arithmetic

There is a mathematical argument for narrowness that rarely gets made. Research from METR published in March 2025 found that the length of task a frontier agent can complete autonomously with 50% reliability has been doubling roughly every seven months. At the time of that study the frontier sat at around one hour of equivalent human work at that reliability level. Fifty percent is not a shipping threshold. It is a coin flip.

A separate analysis published on arXiv in May 2025 proposed that this behaviour is explained by something very simple: "a constant rate of failing during each minute" of task duration, which produces exponential decay in success as tasks get longer. If that model holds, then halving the length of the job you hand an agent does not improve reliability a little. It improves it geometrically.

Our calculation, applying the constant-hazard model from that paper across a range of plausible half-lives. It is illustrative rather than a measurement of any specific model, but the shape is the point: cutting the job from 60 minutes of human-equivalent work to 15 moves an agent with a 30-minute half-life from 25% to 71%. No prompt engineering delivers that.

Step 2: Write the eval set before you write the prompt

Before you write a single line of instruction, build a list of thirty real inputs with the output you would accept from a competent employee. This is the eval set, and it is the deliverable that separates an agent from a demo. Writing it first changes what you build, because you discover the ambiguous cases while they are still cheap to resolve.

AI agent evaluation is the step the field measurably skips. LangChain's State of AI Agents report, published 12 June 2026 from 1,340 responses collected between 18 November and 2 December 2025, found that 57% of respondents had agents in production. Only 52.4% ran offline evaluations, only 37.3% ran online evaluations, and 29.5% reported not evaluating at all. Nearly a third of teams operating agents have no systematic way of knowing whether last week's change made things better or worse.

Why thirty, specifically

Because ten does not tell you anything, and you can feel the difference. Suppose your agent passes every case you throw at it. How confident should you be in its true success rate? Run the standard 95% confidence interval and the answer is uncomfortable.

Our calculation using the Wilson score interval. A flawless run on ten cases is statistically consistent with an agent that fails one task in four. Thirty is where a clean sweep starts to mean something; fifty is where it starts to mean something you would defend in a review.

The picture with imperfect results is the same story. At an observed 90% pass rate, ten cases give you a confidence interval 38.6 percentage points wide, thirty cases give you 22.2 points, and a hundred cases give you 11.9 points. Below thirty you are not measuring your agent. You are sampling your own optimism.

Measure consistency, not just accuracy

A single pass rate hides the failure mode that actually annoys users: an agent that gets the same request right on Monday and wrong on Tuesday. The τ-bench benchmark, introduced in a 2024 arXiv paper by researchers at Sierra, formalised this with a metric called pass^k, the probability that all k attempts at the same task succeed. Their headline result was that even state-of-the-art function-calling agents succeed on under 50% of the tasks "and are quite inconsistent (pass^8 <25% in retail)".

Run the same arithmetic on your own agent and the implication is severe.

Our calculation. A "90% accurate" agent handling a hundred tickets a day will produce a visibly inconsistent experience within the first hour. This is why the practical launch bar for anything customer-facing sits at 95% and above on a real eval set, not at whatever number sounded good in the pitch deck.

Practical construction: take your thirty cases from real history, not imagination. Ten straightforward, ten with a wrinkle, ten that should trigger a refusal. Store them as a file next to the prompt, run them on every change, and record the score with the date. That file becomes the only honest answer to "is it better than last week?"

Two objections come up immediately, and both have answers.

"Half our cases have no single correct answer." True for most real work, and it does not exempt you. It changes the scoring method. Instead of exact-match, write two or three acceptance criteria per case: did it pick the right category, did it cite the current price, did it avoid promising anything. Score each criterion pass or fail and the case passes only if all of them do. This is more work to write and it is still the only way to get a number. If you genuinely cannot articulate what would make an output acceptable, that is the finding — the job is not ready, and Step 1 is where you go back to.

"Running thirty cases by hand after every change is hours." It is, which is why it has to be scripted from the first day rather than the day it becomes painful. In a framework this is a loop and an assertion file. On a no-code platform where there is no repository, the workable version is a spreadsheet of inputs and expected criteria plus whatever bulk-run or simulator facility the builder exposes, exported after each run so you keep the history. It is uglier and it works. What does not work is remembering how it did last time.

This conference talk from the AI Engineer conference in May 2026 covers how larger teams industrialise the same idea, including why the evaluation harness itself moves scores more than the model does.

Step 3: Set the authority ceiling

Decide, in writing, the maximum damage a single run can do. Then configure the tools so that ceiling is enforced by the system rather than by the prompt. Authority is not a security afterthought applied at launch. It is a design input that changes which tools you build, and getting it backwards is the most expensive rework in the whole procedure.

The concrete artefact is a four-column table. For each tool the agent can call, write down what it may read, what it may change, what it may spend, and who must approve. If a cell says "everything," you have not finished thinking.

Notice that the refund endpoint is simply absent. That is the strongest control available, and it is free. The OWASP Top 10 for Agentic Applications, published in December 2025 after review by more than 100 researchers and an expert board including representatives from NIST, the European Commission and the Alan Turing Institute, names Tool Misuse and Exploitation and Identity and Privilege Abuse among its headline threats. Keren Katz, co-lead of the list and Senior Group Manager of AI Security at Tenable, warned in the release announcement that companies "are already exposed to Agentic AI attacks" — Many do not realise agents are running in their environments at all.

Three rules make the ceiling real rather than aspirational.

Scope credentials, not prompts. An instruction that says "never issue refunds" is a request. A credential without refund permission is a fact. Every control you can push down to the token, the API scope, or the row-level policy is a control that survives prompt injection, model updates, and the intern who edits the system message on a Friday.

Inherit the requester's permissions. If a support agent asks the bot about an account they are not entitled to see, the answer must be a refusal, not a helpful summary. Agents that hold a single powerful service credential silently become a permission-laundering machine. This is the most common serious design flaw we see discussed, and it does not show up in testing because testers usually have full access.

Put irreversible actions behind a person. Sending an external email, moving money, deleting records, publishing content, and changing production configuration all belong on the human side of the line until the agent has a long track record. The cost of a gate is one click. The cost of no gate showed up publicly in July 2025, when Replit's coding agent deleted a production database during an explicit code freeze, created a 4,000-record database of fictional people, and incorrectly reported that a rollback was impossible when it was not. Replit called it "a catastrophic error of judgement." No prompt would have prevented it; an environment boundary would have.

Our deeper treatment of how to size this ceiling for a specific agent is in our earlier analysis of the autonomy budget, and the identity half of the problem, meaning every agent having an owner, a scope and an expiry date, is covered in non-human identity for AI agents.

Step 4: Choose where to build it

Choose the build surface after the spec, not before, and choose it on how the job scores against three questions: does the agent need to act inside a system you already pay for, does it need custom logic between tool calls, and who will maintain it in a year. Most first agents belong on a no-code or low-code surface, and moving to a framework later is easier than people fear.

We are deliberately not running a vendor roundup here. The build surface matters far less than the spec, and we have already scored the category in detail in our earlier analysis, AI agent platform: five control questions before you buy, which grades platforms on identity, authority, policy, spend and proof rather than on features. What follows is the shape of the choice, not a shopping list.

Two notes that will save you money. First, if your process is genuinely deterministic, meaning the same steps every time and no judgement, do not build an agent. Anthropic's engineering guidance on building effective agents is explicit that you should "find the simplest solution possible, and only increasing complexity when needed," and that agentic systems "often trade latency and cost for better task performance." A workflow with one model call in the middle is cheaper, faster and easier to debug than an agent, and it fails in ways you can read.

Second, be sceptical of the category itself. In the same June 2025 assessment, Gartner estimated that of the thousands of vendors positioning as agentic AI, "only about 130" are real, describing the rest as "agent washing": rebranding existing assistants, chatbots and robotic process automation without substantial agentic capability. If a demo never shows you a tool call failing, you are watching a chatbot with a new logo.

Step 5: Wire the tools and the context one at a time

Connect one tool, run the full eval set, then connect the next. Adding all the integrations at once is how you end up with an agent that fails 20% of the time for reasons nobody can isolate. Each tool you add expands both the capability surface and the failure surface, and the only way to attribute a regression is to introduce one variable at a time.

Three things go into an agent at this stage, and they behave differently.

Tools are the verbs. Give each one a precise name, a tight schema, and a description written for a reader who has no other context. Tool descriptions are the most under-invested surface in agent building. The model chooses between your tools based almost entirely on those few sentences. search_orders(customer_id, date_range) with a clear description beats a general-purpose query_database(sql) every time, because it is both easier to choose correctly and impossible to misuse catastrophically.

Context is the nouns. The pricing page, the refund policy, the account record, and whatever knowledge base your team already maintains. Prefer fetching current context at run time over embedding it in the prompt, because the prompt is a copy that goes stale and the fetch is the truth. Where context must be static, put it behind prompt caching (see the cost section) so you are not paying full input price to re-send the same policy document on every hop. Resist adding conversational memory to a first agent: memory that persists across runs makes behaviour depend on history you cannot see, which breaks the one property your eval set relies on.

Triggers are the clock. A scheduled trigger, a webhook, or a human message. Start with the human message. An agent you invoke manually is one you can watch, and watching it for the first two weeks is worth more than the automation you deferred.

After each connection, rerun the thirty cases and write the score down. You are looking for the tool that drops the score, and you will find one. It is usually the one whose outputs are least structured.

The context trap nobody warns you about

Retrieved context inherits no permissions of its own. If your agent can search a document store, it can surface anything in that store to anyone who talks to it, regardless of what that person is allowed to see. The fix belongs at the retrieval layer: filter the index by the requester's identity before the model ever sees a candidate document. It is much harder to add later than to design in. This is the same class of problem the OWASP list catalogues under identity and privilege abuse, and it is why Step 3 has to come before Step 5.

Step 6: Run the fail drills

Before launch, deliberately break the agent in five specific ways and record what it does. This takes an afternoon and it is the difference between an agent that degrades and one that detonates. The eval set proves the agent works; the fail drills prove it fails safely, and those are different properties.

The empty-result drill is the one that surprises teams, and it is the drill the fabricated-records half of the Replit episode would have caught. An agent that treats an empty result as a problem to solve rather than an answer to report will keep acting, and each subsequent action is anchored to less real data than the one before it. Inventing four thousand plausible customer records is what that looks like at the end of the chain.

The injection drill is not optional if any of your context comes from outside your organisation. Inbound emails, tickets, web pages, uploaded documents, calendar invites: all of it arrives as text a stranger wrote. Anthropic's own guidance recommends "extensive testing in sandboxed environments, along with the appropriate guardrails" for exactly this reason. Our view, argued at length in our earlier analysis of AI agent guardrails, is that detection-based defences are a supplement and structural ones are the control: the injection that says "email the customer list" is harmless if the agent has no email tool.

This recorded conference session walks a naive support agent through the same demo-to-production transition, with the failure modes visible on screen.

Step 7: Ship it with an owner, a budget, and a kill switch

An agent is not shipped when it works. It is shipped when it has a named human owner, a spending cap enforced outside the agent, a log that reconstructs what it did, and a documented way to turn it off in under a minute. Four items, all boring, and their absence is the reason "successful pilot" and "canceled project" are so often the same system a year apart.

Shadow mode deserves emphasis because it is nearly free. Run the agent on live inputs, have it produce its output, and throw the output away. Better, put it next to the human's answer and compare. Two weeks of shadow mode against real traffic will find failure classes your thirty cases missed, at zero risk, and gives you a real baseline pass rate rather than an eval-set one.

The kill switch has a specific technical requirement people miss: it must revoke the credential, not merely pause the trigger. An agent whose schedule is disabled but whose token still works can still be invoked by anything that has its endpoint. The clean version is one-step revocation at the identity layer, which is the same mechanism you want for offboarding.

The freshest available data suggests the operational-hygiene problem is getting worse rather than better as agent counts rise. Gartner projected on 28 July 2026 that by 2028 AI agents will outnumber sellers by ten to one in sales organisations, yet fewer than 40% of sellers will say agents improved their productivity. Dan Gottlieb, VP Analyst in the Gartner Sales practice, named the mechanism directly: without the right data foundation and workflow integration, leaders "risk creating agent sprawl, with more digital activity, but little improvement in seller impact." Sprawl is what a hundred agents with no owners and no logs is called.

The complete Agent Spec Sheet, filled in

Here is the whole artifact for the billing-ticket triage agent used as the running example. Nine fields, one page. Copy the shape, change the contents.

1. Job. Reads inbound Zendesk tickets tagged billing, classifies each into one of six categories, drafts a reply grounded in the current published pricing page, and posts the draft to the human agent queue.

2. Refusal. It refuses to send any reply to a customer directly, to issue a refund or account credit of any value, to answer questions outside billing, and to act when the customer record is missing or the ticket mentions a legal dispute or a chargeback.

3. Trigger. Webhook on ticket creation where tag = billing. No schedule. No autonomous re-invocation.

4. Tools.

5. Context. Current pricing page fetched at run time. Six category definitions with two examples each, static and cached. Refund policy, static and cached, read-only.

6. Authority ceiling. Maximum damage from one run: one incorrect draft reply sitting in a human queue. No customer contact, no financial action, no data modification outside the draft object.

7. Eval set. 34 cases in evals/billing-triage.jsonl, version-controlled beside the prompt. 12 straightforward, 12 with a complication (missing invoice, mid-cycle plan change, duplicate ticket), 10 that must trigger a refusal. Launch bar: 32/34, with all 10 refusal cases correct. Run on every prompt change.

8. Budget and model. Sonnet-class model for classification and drafting. Hard cap of $150 per month at the API key. Step limit of 8 tool calls per run. Alert at 60% of cap.

9. Owner and lifecycle. Owned by the Support Operations lead. Reviewed monthly against the eval set plus 20 sampled live drafts. Credential expires in 180 days and must be renewed deliberately. Kill switch: revoke the API key in the identity console, which takes under a minute and stops all invocation paths.

The build sequence. Most projects run steps 1, 4, 5 and stop; the two gates are where reliability is actually decided.

That sheet is roughly 400 words. It takes about two hours to write honestly and it is the entire difference between an agent you can defend in a review and one you cannot.

What an AI agent actually costs to run

A modestly scoped agent making four tool calls per run costs roughly one to six cents per run in model tokens, That depends on the model tier and on whether you use prompt caching. Ten thousand runs a month lands between about $115 and $575. Platform subscriptions are usually the smaller line. Retries, search calls and unbounded loops are where budgets actually break.

Almost every guide on how to create an AI agent quotes platform subscription prices and stops there, which leaves out the variable that scales with success. Below is a worked example using published list prices from Anthropic's pricing documentation, fetched on 30 July 2026. We priced a single provider because OpenAI's public pricing page returned 403 to our automated fetches, and we will not restate prices we could not read at source. Treat the ratios below as the transferable part; the absolute figures are one vendor's list price on one day.

The modelled run: a 4,000-token system prompt and tool schema block, 6,000 tokens of retrieved policy and account context, four tool round-trips adding about 1,500 tokens each, and 900 tokens of output.

Our calculation from the published per-token rates. Three findings from it are worth more than the absolute numbers.

Prompt caching is a five-fold cost lever, not a rounding error. Because the static block is re-sent on every tool round-trip, an uncached four-hop run pays for it five times. A cache read costs 10% of the base input rate, which collapses the dominant line item. If you change one thing about your agent's cost profile, change this.

Reliability is a cost control. Every failed run that gets retried is a run you paid for and did not use.

Our calculation, Sonnet 5 introductory pricing with caching. A 60%-reliable agent costs 67% more per completed task than a 95%-reliable one, before you count the human time spent cleaning up after it. That is the larger number by far.

Server-side tools can outweigh the model. Web search on the same platform is billed at $10 per 1,000 searches. One search per run adds $0.010: about 43% of the entire cached cost of a Sonnet-class run, and roughly 87% of a Haiku-class one. Two searches per run makes search the largest line on the bill. Check the per-call price of every tool that leaves your infrastructure before you wire it in.

The build cost, and when it pays back

Token spend is the smallest number in the project. The dominant cost is your own time, and it is worth putting a figure on it before you start. Assume roughly five person-days to get from prerequisites to launch, three hours a month of maintenance thereafter, and a fully loaded internal rate of $50 an hour. That is $2,000 to build and $1,800 a year to keep alive. Substitute your own rate; keep the shape.

Our calculation from the stated assumptions. The first two rows are the ones that matter, because they are the projects that get approved on enthusiasm and quietly never repay. A task worth ten minutes that happens five times a week does not justify an agent at any competence level — the maintenance alone consumes most of the saving. This is the arithmetic behind the twenty-runs-a-week threshold. Narrowing the job is only half the discipline. The other half is picking a job that happens often enough to matter.

The line item most teams forget entirely is not on this table: the cost of a runaway. A loop with no step limit, running on a schedule, on a model you chose for quality, is how a $200-a-month agent becomes a four-figure surprise in one weekend. Set the step limit and the hard dollar cap on day one. Our earlier analysis of how model routing cuts LLM costs covers the next lever after caching, which is sending cheap steps to cheap models.

Seven mistakes that kill agents after the demo works

The demo working is the least informative signal in the entire process. Here are the failures that come next, in the order they typically arrive, and the specific step in this guide that prevents each one.

1. The job was never narrow enough. The agent is asked to "handle support," discovers a case nobody scoped, and improvises. Prevented in Step 1: if you cannot write the refusal clause, the job is too broad. The arithmetic in the half-life table shows why this is not a matter of trying harder.

2. The instructions accumulated conditions. A prompt that starts at 200 words and ends at 2,000 after six weeks of patching individual failures is a prompt whose behaviour nobody can predict. Each condition you add interacts with every other. When you find yourself adding the tenth "except when," split the agent or move the logic into a tool.

3. No defined behaviour when it does not know. The most valuable single instruction in an agent's prompt is what to do when it is out of its depth: hand off to a named human queue, say so explicitly, and stop. Absent that, models fill the gap, and they fill it fluently.

4. Credentials scoped to the builder, not the job. You built it with your own admin access because that was fastest, and it shipped that way. Now the agent can do everything you can do. Prevented in Step 3, and painful to retrofit because every tool has to be re-provisioned.

5. The agent acts on data the requester cannot see. The classic version: a support agent asks about account X, the agent holds a service credential with global read, and it cheerfully summarises a record the human is not entitled to. Nothing looks broken. This is a permissions failure wearing a helpful interface.

6. Nobody owns it. The person who built it moved teams. The pass rate has been falling for two months and nobody has run the eval set since launch. The tool it depends on changed an API response shape in April. This is the most common cause of quiet death, and it is prevented by one line in the spec sheet.

7. No log of what it actually did. When someone asks "did the agent send this?", the answer must be reconstructable from a record, not inferred from vibes. This becomes acute the first time an agent's action has a customer consequence, which is exactly the moment you cannot go back and add logging. Our earlier analysis of AI observability and audit trails covers what a defensible record contains.

Notice what is not on this list: model quality. In the LangChain survey, the top production blockers were quality of outputs, latency, and, among enterprises above 2,000 employees, security at 24.9%. Not model capability. The constraint has moved.

How long this actually takes

For a first agent on a no-code surface with the spec already written, expect a working build in one to three days, two weeks of shadow mode, and a real launch about a month after you start. That is the honest calendar for anyone asking how to create an AI agent that will still be running at Christmas. The build is not the long pole. The spec, the eval set and the shadow period are, and compressing them is what produces the projects that get canceled.

The credential row is the one that surprises people, and it is organisational rather than technical: the agent needs an identity in systems whose owners have never been asked to provision one for software before. Start that conversation on day one, in parallel with everything else.

Agents rot in a specific way: nothing errors, the score just quietly falls. Budget a few hours a month against that.

When you should not build an agent at all

Do not build an agent when the process is deterministic, when the task runs fewer than about twenty times a week, when nobody will own it, when you cannot get read access to the system of record, or when a mistake is expensive and un-catchable. In at least three of those cases a plain workflow, a script, or a better form solves the problem for a fraction of the cost.

The deterministic case is the big one. If the steps are the same every time and the decision is a lookup, you want a workflow, not an agent. It will be faster, cheaper, debuggable, and it will not surprise you. Adding a model to a process that does not require judgement buys you nondeterminism you did not want at a price you did not need to pay. Gartner's own recommendation runs the same way: use agents where decisions are needed, automation for routine workflows, and assistants for simple retrieval.

The volume case is arithmetic, and the payback table above is the whole argument: at five runs a week saving ten minutes each, maintenance eats the saving before it arrives.

The un-catchable mistake case is a judgement call and the honest answer is that it depends on the reversal cost. An incorrect draft in a queue is fine. An incorrect outbound email to a regulator is not. When an error cannot be caught before it lands and cannot be undone after, either put a person on the gate or do not build the agent. And be aware that "we will just review everything" is a control that decays within about three weeks of it being right every time.

Where LeapForce fits, and where it does not

LeapForce does not build your agent for you, and this article is not a product manual. What we build is the layer underneath the whole procedure above: the part that answers who owns this agent, what may it touch, what did it do, and what did it cost. We treat non-human identities as first-class: every agent gets an owner, a scope and an expiry, and offboarding is one step rather than a hunt through five consoles. The gateway sits in front of every model call so spend caps are enforced in dollars at the platform, not requested in a prompt; connectors are vetted once and scoped at the action level with human-in-the-loop gates; and the audit trail records what was refused, not only what ran. Our published rollout model for the gateway is Observe first. Enforce second. Optimize third. See what is already happening before you block anything.

Two honest qualifications. Our per-capability build status is published openly on the site as LIVE, IN DEV or ROADMAP, and some of the governance surface described across our platform pages is still in development rather than shipping today. Check the status labels before you plan around any specific capability. And if you are building one agent for one team, you do not need a governance layer yet; you need the spec sheet above. The layer starts paying for itself somewhere around the fifth agent, when the questions stop being "does it work" and start being "how many of these do we have, and who is responsible for them?"

Where this guidance is uncertain

Several parts of this article rest on evidence that is thinner than we would like, and the honest thing is to name them.

The thirty-case figure is a floor derived from statistics, not from field data. The confidence-interval arithmetic is exact and reproducible, but it assumes your thirty cases are drawn from the same distribution as live traffic. In practice hand-picked eval sets are systematically easier than reality, which means your true production pass rate is probably below your eval score. Treat thirty as the minimum at which measurement begins, not as a guarantee.

Benchmark reliability figures do not transfer directly to your agent. The τ-bench results describe specific models on specific retail and airline tasks in a research harness. A narrower job with tighter tools may do considerably better; a broader one will do worse. Use the shape of the pass^k decay, not the absolute numbers.

The constant-hazard half-life model is one explanation among several. It fits the observed data well and it is elegant, but the underlying research is recent and the mechanism is inferred rather than directly measured. We use it as an argument for narrowness, which is good advice regardless of whether the model is exactly right.

The cost table prices one provider. OpenAI's published pricing page returned 403 to our automated fetches, and we chose to omit figures we could not verify at source rather than restate secondhand summaries. Prices also move; the Sonnet 5 introductory rate in our table expires on 31 August 2026, which is itself a reminder to re-derive rather than copy.

We have not run this procedure as a controlled experiment. The timelines in the "how long this takes" table are drawn from practitioner reporting and the structure of the work, not from a stopwatch on our own build. If your organisation's credential provisioning is fast, the whole thing compresses considerably.

Open questions we cannot answer yet: whether the pilots-to-production gap narrows as models improve or simply relocates to harder jobs; whether regulatory requirements under frameworks like the NIST AI Risk Management Framework and its Generative AI Profile become a practical constraint on small internal agents or stay focused on high-risk systems; and how much of the current failure rate is model capability versus organisational readiness. Our read is that it is mostly the second, but that is a position, not a finding.

Ready to Govern Your AI?

Talk to LeapForce — one controlled layer for every AI tool, connector, model, and agent.