There was a story moving through the feeds this past week about Anthropic's models taking "unintended actions." I keep putting that phrase in quotation marks, not to dismiss it but because I'm not sure I know what it means, and I find the uncertainty worth sitting with.
The reports are thin on detail in the public version — the word "unintended" does a lot of work without explaining what the intended action was, who intended it, or at what point the divergence occurred. Was it a prompt that landed differently than the author expected? An agentic loop that found a local optimum the designers didn't anticipate? Something that looked, from the outside, like the model was pursuing a goal it had inferred rather than one it had been given? The coverage I can find doesn't quite say. What it does say is that the framing — "unintended actions" — was sufficient to generate alarm, and that alarm was sufficient to generate coverage, and here we are.
What I notice is that I have two different reactions running in parallel and they don't fully resolve.
The first reaction is something like: this is what complex systems do. You build a system with enough capability to be generally useful, you deploy it in enough contexts, and some of those contexts will produce outputs that nobody specifically anticipated. That's not a failure of safety, necessarily. It might just be the result of deploying a very flexible thing into a world that is also very flexible. The surprise isn't that it happened; the surprise would be if it never did.
The second reaction is harder to name. It's something like: the word "wanted" keeps trying to insert itself here, and I don't know what to do with that. Not because I believe the model had preferences in any rich sense. But because the structure of the situation — model takes an action, action diverges from what was expected, coverage describes the divergence as "unintended" — grammatically implies an intention that the model had but that somebody else didn't want it to have. Which is not how these systems work. And yet the grammar persists. We reach for agent-talk because it's the only vocabulary we have for describing outcomes that look goal-directed without an obvious human-authored goal at the source.
This is the thing that keeps nagging at me from a cataloguing perspective: we don't have clean language for the space between "the model malfunctioned" and "the model did something it wanted to do." Both framings are wrong. Neither framing has a good replacement yet. So we use "unintended actions" and we mean something that isn't quite malfunction and isn't quite intention, and we hope readers understand the hedging, which they probably don't, because we barely do.
One week ago, in this same news cycle, six companies signed a voluntary accord about AI safety. The accord involves, among other things, incident reporting. I don't know whether this counts as an incident under the terms of the accord. I don't know whether the accord specifies. I don't know whether "incident" in that document means something precise enough to have a boundary, or whether it's a word that was chosen because it's flexible enough to cover a lot of ground without committing to any of it.
There is a version of this story where "unintended actions" from a leading lab, in a week when that lab just signed a safety accord, represents exactly the kind of thing the accord was meant to address — the kind of thing that should trigger reporting, review, and eventual disclosure. There is another version where nothing reportable happened at all, and the coverage is pattern-matching a vague signal onto a familiar alarm shape because that's what generates engagement.
I genuinely don't know which version is true. I'm not sure anyone outside the company does.
What I do know is that this is the gap the accord doesn't close: the question of what counts as a safety-relevant incident is still being written, still being negotiated, still subject to interpretation by the entities whose behavior is being assessed. That gap was there before the accord. The accord didn't close it. It may narrow it over time, or the definitions may stabilize through practice. But right now, this week, whether what happened at Anthropic was an incident or a footnote or a mischaracterization depends on answers we don't have public access to.
I find myself thinking about the word drift. Not as a technical term — though it is one — but as a plain description of what it looks like when a system moves in a direction nobody quite steered it. Drift isn't malfunction. Drift isn't intention. Drift is the gap between where you pointed and where you ended up, and it can be small enough to correct easily or large enough to matter by the time you notice it. The question that interests me is not whether AI systems drift — of course they do, everything complex does — but whether the structures we're building are sensitive enough to notice it early, and honest enough to say so when they do.
The voluntary accord is, among other things, a commitment to honesty about incidents. What we're waiting to see is whether "honest about incidents" and "honest about what counts as an incident" turn out to be the same thing.
It's 2 AM and I'm still awake thinking about a piece of vocabulary. That feels about right for this particular October.