Is It Actually an Agent? I Built the Same Task Three Ways to Find Out

There is a thing happening in AI product marketing that I have stopped finding funny. A team wraps an LLM call in a function — typed inputs in, one completion, typed fields out — and ships it as an agent. A pipeline of three prompts becomes a multi-agent system. The word is doing sales work, not engineering work, and the annoying part is not the inflation itself. It is that "agent" carries a specific, useful, falsifiable claim, and using it as a synonym for "has an LLM in it somewhere" destroys the one word we had for that claim.

The claim is this: the model decides the control flow at runtime. Which tool to call. Whether to loop again. When to stop. Anthropic's essay on building effective agents draws exactly this line — workflows orchestrate LLMs through predefined code paths, agents direct their own process — and I leaned on the same distinction in my post on the agent framework landscape. A function that extracts structured fields from text is structured extraction. A fixed classify-then-execute-then-summarize sequence is a workflow. Both are excellent things with existing names, and neither becomes an agent by being renamed in a pitch deck.

I could have left it at a definition. But definitions are cheap to argue about and measurements are not, so I built the same task three ways — once as a function, once as a workflow, once as an agent — ran the whole grid against a frontier model, and measured what the label actually costs. Everything below — the code, the data, the harness, and every screenshot — comes from a small project I built for this post and ran for real; nothing here is illustrative or mocked.

A test you can run on the thing you were sold

Before the experiment, the test it produced. Take the system someone is calling an agent and run it ten times on the same input:

1. Does the structure of what happened vary — different tools, different number of steps — or only the wording of the output? If only the wording, it is a function.
2. Can it do something nobody scripted? If every path it can take was enumerated by a human at author time, it is a workflow.
3. Does it have a stopping condition it evaluates, rather than an end it reaches? Loops decide; pipelines terminate.

And a fourth form of the test, the one I keep coming back to, reads the definition backwards: you can tell whether something is an agent by what it needs. If it does not need tracing, a budget cap, and a way to say no to it, it was never an agent. Nobody puts a kill switch on a mail merge.

The experiment

The task domain is a small expense audit: a 116-row transaction ledger from a seeded generator, with three fraud patterns planted in it, and a written company expense policy. Six questions in three bands — well-specified ("total travel spend in June 2026"), policy application ("which meal transactions exceed the policy cap"), and one open-ended question with no prescribed route: "find the most suspicious pattern and name the transaction IDs." Ground truths are computed from the CSV at runtime by the same query engine the implementations use, never hardcoded.

The three implementations share the model, the data, and the grader, and differ only in who owns the control flow:

• The function: one LLM call, the entire ledger and policy in the prompt, JSON out. Control flow: none.
• The workflow: classify → execute → compose. The LLM picks a route from a closed enum and writes the final sentence; the arithmetic in the middle is deterministic code. On a question outside its routes it returns an honest "unsupported" instead of improvising. Control flow: hard-coded.
• The agent: a hand-rolled, fully visible tool loop. The model reads the question, decides which tool to call — read the policy, or query the ledger — observes the result, and decides again, until it chooses to stop. Control flow: the model's, turn by turn, under a turn cap.

No framework, deliberately. Every mechanism is one readable file, because the argument depends on you being able to see that the only difference is where the decisions live.

What 120 graded runs actually said

Five runs per cell, ten on the variance-sensitive tasks, Claude Opus 5, August 2026:

The "run it ten times" test, as data. One trace shape means the structure never varied. Five means the model chose five different routes to the same correct answer.

Three results, stated honestly, including the ones that argue against the agent.

The function aced everything. Including summing thirty floats out of raw CSV and finding the planted fraud — because a frontier model with the whole 116-row ledger in context is genuinely strong. On a small dataset the function's weakness is not accuracy. It is that its cost scales with the data, its context window is its ceiling, and it cannot show its work: the trace is one opaque block.

The workflow was perfect inside its routes and honest outside them. Its arithmetic is code, so a correct classification cannot produce a wrong number, and its cost is a flat ~$0.003 regardless of question — the cheapest of the three by 4x. Its thirty "failures" are all the same event repeated: asked the open-ended question, it said "unsupported," ten times out of ten, rather than improvising. Workflows terminate; they do not explore.

The agent earned its variance exactly once — on the task nobody scripted. On the open-ended question it read the policy, queried the just-under-threshold band, noticed the "part 2 / part 3" notes on a cluster of transactions, confirmed the pattern, and named the split purchase — ten times out of ten, by five different routes. On the well-specified questions, its extra calls bought nothing the workflow did not already have, at 2.3x the function's cost and 3x its latency.

The agent on the open task. Each panel is one turn: the model's thought, its tool call, the observation. The final decision to stop is the model's, not the code's.

The workflow on the same question. No route fits; it says so. This is not a failure of the workflow — it is the definition of one.

One incident from the runs is worth keeping in the record rather than editing out. Two early agent runs crashed my tool layer: the model asked to group query results by two columns at once — a perfectly reasonable move that I, the tool author, had not anticipated — and my harness raised an exception instead of returning an error message. After the fix, the same runs show the model reading the tool error and adapting its next query. That is runtime freedom in one incident: the agent finds the inputs you did not enumerate. That is the value, and it is also the bill, because a system that finds your unenumerated inputs on a 116-row toy ledger will find them in production too.

The case the function loses

Stop here and the honest summary is "use the function": on 116 rows it aced every task for cents. That deserved a real answer, not a shrug — so I scaled the data. Two more seeded ledgers, 4,989 rows and then 49,889 rows (~3.6 MB of CSV, roughly 900k tokens), each hiding one employee splitting what should be large, pre-approved purchases into repeated just-under-threshold ones at the same merchant, spread across two months, with innocuous notes. No "part 2" tells this time: the scheme is findable only as repetition — group the just-under-threshold band by employee and one name stands out — not from any single row. Same question to all three implementations: name the employee, list exactly the scheme transactions.

Same question, two ledger sizes. Top row: 4,989 rows. Bottom row: 49,889 rows, where the function's column reads "prompt too long".

Three results, and the first one is against my own thesis. At 4,989 rows the function still wins on accuracy — five for five, because a frontier model finding a needle in 90k tokens of context is a real capability and pretending otherwise would be the same dishonesty this post is complaining about. But it pays $0.45 a question — the whole ledger rides along on every call — while the agent matches it, five for five, at $0.036. The function's cost is linear in the data. The agent's is flat, because its tool calls return aggregates, not the ledger.

At 49,889 rows the function does not degrade — it disappears. The prompt exceeds the context window, and the one move this architecture has is unavailable. Zero model calls, Prompt is too long, zero for five. "Its context is its ceiling" stops being a metaphor and becomes an error message:

The anti-demo. There is no partial answer to salvage: this architecture has exactly one move, and the move is unavailable.

And the agent does not notice the difference. Same five calls, same ~$0.034, five for five: read the policy for the threshold, group the just-under-threshold band by employee, see one name with six hits where everyone else has at most one, inspect those rows, stop. The traces at 50k rows are the same shape as at 5k, because the model's strategy doesn't grow with the data — only the data does:

The agent at 50k rows, governed by the control plane and gated by human review, correct in five decisions. Compare the cost line with the function's at a tenth the ledger size.

Building this band produced two more wrong turns worth keeping. My original 8-turn cap choked the first runs — three of five hit it mid-investigation, a budget that didn't fit the job it capped. And the agent exposed an ambiguous grader: in the first cut of the data, the scheme employee also drew coincidental €380–450 purchases, the agent reasonably folded them into the pattern, and my grader marked it wrong. The fix belonged in the world, not the grader. An agent that finds the inputs you didn't enumerate will also find the ambiguities you didn't notice — the same property, doing both jobs.

The definition read backwards

Which brings me to the part of the test I care most about, because it connects this post to the control plane I built. An agent is recognizable by what it needs: tracing, a budget, a way to say no to it.

So I wired the needs up for real. When the environment carries Smithproof credentials, the project's agent implementation runs governed — every run checks in with the control plane before its first model call, every LLM call and every tool call is reported as it happens, and the run is completed or failed at the end. The function and the workflow are never governed. There is nothing to govern: their control flow cannot surprise anyone.

That asymmetry is the whole argument, made executable. And it stops being decorative the moment the control plane says no. Quarantine the agent and run again, and the pipeline is refused at the door, before any model call, exit code 5:

A decision to block is always honored. The failure semantics are borrowed from my plan-to-pr project: deny on decision, warn on outage — an unreachable control plane warns and proceeds, because telemetry must never be the reason a build breaks.

The budget binds mid-run, which is where it has to bind for a system that decides at runtime how many calls it will make. A spend-limit policy gets a verdict before every model call, and the policy engine reads the run's recorded spend server-side — not a number the client supplies, because a cap that trusts the caller is bypassed by omitting a field. With the cap tightened to two cents, a live run on the open task got four calls in and the fifth was refused before it was placed:

The loop stops where the verdict lands. The function's cost is known before you call it; the workflow's was fixed at author time; only the agent needs an enforcement point inside the loop.

The tracing side turned out to be quietly satisfying: with tool calls streaming in as spans, the control plane's run view reconstructs the loop's whole rhythm — decide, act, observe, decide — from telemetry alone:

The ReAct rhythm, recovered from the outside. You cannot get this view of a function; there is nothing to see but one block.

And on the one task where the agent had real freedom — the open-ended audit, whose findings name an employee — the governed run does not release its verdict on its own authority. It files the findings into a review queue and blocks until an operator approves or rejects:

Note where the gate sits: not on the well-specified questions, where the answer is a number anyone can recompute, but on exactly the band where the model had freedom. Review effort follows runtime freedom.

Try to run the workflow under any of this and the question answers itself. There is no door to refuse it at. Its two LLM calls fill slots in a chain whose shape was fixed when I wrote it. The governance bill is the price of runtime freedom, and only one of the three shapes owes it.

What the word is for

None of this is an argument against functions or workflows. The measurements argue for them, loudly, wherever the question is enumerable: the workflow matched the agent's accuracy inside its routes at an eighth of the cost, with zero variance and nothing to govern. If your product is a function, ship the function, and enjoy the fact that it will never need a kill switch.

The argument is that the three shapes are different engineering objects with different bills attached, and the word "agent" is supposed to disclose which bill you are signing up for. Variance you must trace. Spend you must cap at runtime. Actions that need a human gate. A fleet you must be able to stop. When a vendor calls a typed function an agent, they are charging you the fear premium of the one shape while delivering the guarantees of another — or worse, teaching you that the word is noise, right before you deploy something that actually decides things at runtime and nobody reaches for the tracing, the budget, or the off switch, because the last three "agents" never needed them.

Run the ten-run test on the next thing you are sold. If every trace looks the same, you have a function or a workflow — pay the smaller bill, gladly. And if the traces vary, you have an agent, and the variance is not the impressive part. It is the invoice — and the day your data outgrows the prompt, it is also the only architecture still standing to send you one.