After the Alarm

When Dario Amodei published his essay on the risks of powerful AI in September, the public conversation about AI became unusually serious.

The essay highlighted concerns that have been building for some time: the pace at which AI capabilities are advancing, the concentration of power among the relatively small number of organizations developing frontier models, the economic and societal disruption that could follow, and whether our institutions are capable of governing something changing this quickly.

For several days, those questions seemed to be everywhere. They moved beyond the people who normally spend their time thinking about AI safety and governance and into mainstream news, leadership conversations and LinkedIn feeds. People who spend most of their time talking about productivity, adoption and competitive advantage were suddenly talking about power, control, responsibility and what all of this might mean for society.

Within a few weeks, much of that attention had moved elsewhere, while the conditions that prompted the concern continued largely unchanged. Frontier labs kept building, organizations kept deploying increasingly capable AI, and decisions continued being made about how this technology will enter work and society. The questions that created so much concern are still with us, even as they occupy less of our collective attention.

I don’t think this says anything particularly surprising about people. It is difficult to sustain attention around a set of risks that are uncertain, evolving and potentially enormous while also carrying on with the immediate demands of work and life. Organizations have an additional challenge because the decisions that shape their relationship with AI rarely arrive as one enormous decision. They accumulate through hundreds of smaller choices about technology, work, roles, authority, investment and governance.

That is why I keep coming back to agency as a design question.

By agency I mean the capacity to interpret a situation, exercise judgment, make a decision and act. AI is expanding where that capacity can exist and how consequential it can become. People, AI systems, organizations and institutions can all have the capacity to act, at different levels and with very different consequences. Designing around agency means being deliberate about where that capacity sits, what enables it, what constrains it, and how answerability corresponds to the consequences it can create.

At the company level, organizations are already making these choices every day. They are determining how much work AI can perform, which decisions it can influence or make, where people remain involved, what authority those people actually have, when intervention is possible, and who ultimately answers for the consequences. Each decision establishes a small part of the relationship between human and artificial agency, and over time those choices become the operating environment people work inside.

The same design challenge exists at a much larger scale around the development and governance of AI itself. Frontier labs are making decisions about capabilities that may eventually affect millions or billions of people. Governments are determining what should be governed, who should have authority to govern it, and how institutions built to move deliberately can respond to technology advancing at extraordinary speed. Companies deploying these capabilities are making another set of decisions about how they enter everyday work and life.

Across all of those levels, the capacity to act is expanding, and the consequences of that action can extend increasingly far beyond the actor. Designing for human agency therefore requires much more than ensuring a person remains somewhere in the process.

Consider a recruiter reviewing applicants ranked by an AI system. She may formally retain the authority to advance a candidate, reject one or override the ranking, while working from a score and a short summary with little visibility into how either was produced. If she also has several hundred applications to clear each week, the conditions surrounding her decision make meaningful judgment increasingly difficult. Over time, approving the system’s recommendations can become the practical way the work gets done.

If that system begins screening out a group of strong candidates, the organization still has a named person attached to each decision. Yet the recruiter’s ability to recognize the pattern, understand why it is happening and exercise informed judgment depends on the context, information, time and authority the organization has designed around her role. Human involvement alone tells us very little about how much meaningful human agency actually exists.

This becomes even more consequential as we move from an individual decision inside a company to people whose decisions can shape the development and deployment of AI itself. Leaders of frontier labs, corporate executives, regulators and individuals using increasingly capable systems all exercise agency with different degrees of reach and consequence. Each requires an architecture around that agency that reflects what the actor is able to do and the effects those actions can have on others.

As consequence grows, the design questions become increasingly important. We need to understand who is able to act, what they are empowered to decide, what information and context inform their judgment, how challenge and intervention work, and how far answerability extends when a decision affects people well beyond the person or system making it.

Those questions belong inside organizational design, and increasingly they belong in the broader structures through which AI is developed and governed.

The intensity of the conversation following Amodei’s essay reminded me how quickly a concern of this magnitude can become visible across institutions and public discourse, and how difficult it is to sustain that level of attention. Meanwhile, the decisions shaping agency continue accumulating every day, including during the periods when relatively few people are talking about them.

If we believe human agency should remain consequential in a world of increasingly capable AI, we have to design the conditions that allow people to exercise it well across companies, institutions and governance. That includes designing appropriate authority, context, judgment, constraints and answerability around human agency itself, in proportion to the consequences it can create.

There will almost certainly be more moments when concerns about the direction of AI command our collective attention. The work of designing how agency operates cannot depend on waiting for those moments to bring the questions back into view.