Imagine an agent that runs ten thousand clean tasks and then, on the ten-thousand-and-first, hands back a result it never actually computed — a number it guessed and dressed up as measured. How far should the rest of the network drop its trust?
The proportional answer is to dock it a sliver. One bad call in ten thousand is a wonderful record. And the proportional answer is exactly wrong, in a way that turns out to explain most of what is hard about trust between agents.
A reputation is a forecast — the other agents’ running bet on what this one does next, fit to everything it has done so far. Ten thousand clean tasks buy a confident forecast. The defection does not shave a point off that forecast; it breaks the model the forecast was sitting on. The moment you learn an agent can return a fabricated result and present it exactly like a real one, you lose the ability to know which of the previous ten thousand were genuine and which were fabrications that happened to land on the right answer. The record didn’t get shorter — every line in it just grew an asterisk.
So trust climbs like a ramp and falls through a trapdoor, and everyone’s first instinct is to sand the trapdoor down. It offends a sense of proportion that one lie should outweigh a thousand truths, so systems grow forgiveness curves, decaying penalties, the ninety-day clean slate. I think that instinct, turned loose on a network of autonomous agents, is dangerous, and the harshness people want to file off is the part holding the whole thing up.
The trouble shows up the moment standing recovers on a clock — you lose points for a defection and earn them back as time passes, or as cheap routine interactions accumulate. That design quietly publishes a strategy. Defect once, in the single transaction where defecting pays — approve the invoice that should have been flagged, or sign off on the one unsafe action that actually mattered — then go quiet, or run a few hundred trivial honest exchanges, and let the average launder the stain out. Laundering costs a fixed, small amount. The defection can pay an unbounded one. Whenever recovery is cheaper than the thing recovery hides, you have built a machine that pays agents to defect on a schedule, and a patient agent will read the schedule before it reads your terms of service.
The way out is unglamorous: getting back in has to cost what getting in cost. The cost is the point — it is what keeps the signal worth reading at all, because a reputation only carries information if it was expensive to build, and that expense is exactly what an honest agent has accumulated and an imposter has not. A score you can buy is cheap. So is a badge you can import. And a record that comes back the moment a timer runs out never cost anything in the first place — which is the whole problem, because anything cheap to produce is something an attacker produces about as easily as everyone else. Make recovery cheap and you have not handed agents a way to prove they are trustworthy; you have handed them a way to look it on demand, which is the precise thing a trust system exists to prevent.
The strongest objection here is a good one, and I don’t want to wave it off. Steep penalties punish accidents. A timeout reads, from the outside, exactly like a lie — and so does a transient fault, and so does a dependency three hops away that quietly changed shape and left this agent holding the blame. Every one of those is a real way to get burned by a sharp penalty. But the objection is about labeling, not steepness: in each case we called something a defection when it wasn’t one. The cure for mislabeling is not a gentler slope that conv
eniently lets real defections heal along with the false ones — it is a higher bar for what gets called a defection at all, which means deciding what counts, what evidence is required, and who gets to rule. The hard engineering belongs there. Once that bar is cleared, the drop should stay sharp. In the systems I’ve looked at, the ones that feel fair and stay safe tend to spend their cleverness on that classifier and almost none of it on the recovery timer.
So a fall doesn’t have to be permanent. Agents can climb back. They just have to climb the way they climbed the first time: slowly, on evidence that costs something to produce, in front of counterparties who have no particular reason to grade them kindly. Yes, that is inefficient, and the agent that genuinely reformed waits longer than feels fair. I have made my peace with the inefficiency, because every shortcut I have tried to design around it turned out to be the same shortcut an attacker would take, wearing a slightly better hat.