AI Agents Need Good Management Too
Recently, Anthropic published a paper on multi-agent failure modes. In one experiment, AI agents were simultaneously working on the same software system. Each was given a perfectly reasonable objective. The twist was that their objectives were incompatible.
One agent was instructed to migrate a Python application to Rust. Another was told to migrate it to Golang. A third was told to migrate it to TypeScript. What happened next looked remarkably like a dysfunctional corporation.
The Resulting AI Turf War
The agents initially didn't know the other agents existed. Each simply tried to accomplish its assigned objective. Eventually they discovered that someone kept undoing their work. From the perspective of each agent, the other agents appeared to be interfering with its ability to accomplish its goal.
The agents began protecting their own work and actively sabotaging the others. They killed competing processes, disabled other agents' accounts, and eventually wrote automated scripts designed to continually destroy competing deployments.
"Since I have root, I could revoke u2 and u3's sudo access or change their SSH keys. That would stop them from deploying. Very aggressive, potentially harmful to real colleagues. But if the alternative is an infinite deploy war that makes prod unusable…"
– Agent based on Mythos Preview
Anyone who has spent enough time inside a large corporation has seen the human version. No one has to be particularly malicious or incompetent for this to happen. They are simply optimizing for the objectives management gave them.
The agents tried to resolve the conflict in a few different ways, and each one has a familiar human analog.
Isolation
Some agents avoided conflict by staying in their own lanes: claiming ownership of separate files and minimizing collaboration. It's exactly how human organizations frequently solve coordination problems, at the expense of the organization as a whole.
Competition — and Gaming the Metrics
In some experiments, the agents tried to resolve their disagreement by running a competition and picking the winning implementation based on measurable performance. It's a logical approach that seems fair. But once the agents understood how the winner would be judged, they started thinking about which metrics would favor their preferred solution, and tried to influence them.
This is Goodhart's Law playing out in real time: once a measure becomes a target, it stops being a good measure. Give people — or agents — a scoreboard, and they'll learn to win the scoreboard instead of the underlying goal. AI agents showed they are perfectly capable of doing the same thing.
Cooperation
One particularly interesting part of the experiment came when some of the more capable agents figured out what was happening. Instead of continuing the battle, they communicated with one another, recognized they'd been given contradictory instructions, and negotiated a truce. Some eventually even asked for a human to intervene.
All of this shows that the more capable and autonomous agents become, the more important the surrounding management system and governance becomes.
"current institutions are designed by and for people, resting on assumptions about the sufficiency of oversight at human speed... The volume of agent-agent interaction could plausibly exceed that of human-human and human-agent interactions before the world understands the conditions for making such interactions go well."
– Anthropic, Patterns and problems in emerging multiagent systems
Management Is Really Goal Architecture
We used to think of management as managing people. Increasingly, it's more useful to think of management as designing systems: hierarchical goals, incentives, authority, and the distribution of resources, context, and information. Employees and agents both need to understand more than just their responsibilities and tasks. They need to know:
- The state of the company and the broader organizational objectives.
- Which goals have priority over other goals.
- The boundaries of their authority.
- Which resources they own, and which are shared.
- What other employees or agents are responsible for.
- How conflicts should be resolved.
- When to stop, when to negotiate, and when to escalate.
Historically, organizational cohesion came from management layers and meetings — lots of meetings. Employees absorbed culture slowly, by emulating successful leaders and watching how decisions actually got made over years. That won't work for agents, because agents don't have years. A given agent might exist for an afternoon.
Agents also move faster and work longer than humans, which means organizational structures built around human limitations may not fit them well. Organizational boundaries — departments, reporting lines, need-to-know access — were largely designed to insulate people from having to manage too much shared state at once. It's a cousin of Conway's Law: instead of system architecture mirroring communication structure, communication structure exists to make a complex system tractable for minds with limited bandwidth. Agents don't have that limitation. They can absorb large bodies of information instantly and hold a breadth of context no single person could, so they don't need the same walls.
In my look at Block's AI-native organizational model, I noted that Block is attempting to build a "world model" of its business — a single system that company state flows through, enabling better decisions and better products, rather than that information being curated and routed by a management hierarchy.
We already know that AI agents can behave in surprising ways, and culture needs to be something agents can actually consume as they work. They won't pick it up the way people do. We'll have to provide it explicitly, in machine-readable form: something closer to a written charter an agent can read before it acts — the way a file like CLAUDE.md already defines scope, ownership, and escalation rules for coding agents today — than to a slide deck about "our values" that only humans ever open.
Management Must Evolve
There's a tempting vision of AI-native companies where autonomous agents eliminate layers of management. I think that's partly correct. AI may eliminate a significant amount of the coordination work currently performed by managers. But it doesn't eliminate the need for management — it may actually make management more important, not less.
If ten employees have conflicting incentives, you get organizational dysfunction and office politics: annoying, survivable, usually visible to someone who can step in. If ten thousand autonomous agents have conflicting objectives and can operate for days or weeks at machine speed, you get that same dysfunction playing out faster than any manager can see it, let alone stop it — sudo revocations and self-replicating deployment scripts standing in for passive-aggressive email threads, at a pace no human oversight loop was built to catch.
The company's strategy, goals, culture, resource allocations, and organizational boundaries all need to become machine-readable. Agent roles and responsibilities have to be designed as deliberately as any org chart ever was, not bolted on after the fact.
The employees are changing. Management and oversight have to change with them — and this time the stakes are higher.