Neutral, Not Park

Last night, I tried twice to record a short video about Dario Amodei’s latest essay, We Must Pace the Frontier, and I couldn’t do it.

It wasn’t because I didn’t have anything to say. If anything, I had too much to say. I had already spent about 30 minutes recording my thoughts after reading through his proposal. I knew what concerned me, I knew what I agreed with, and I knew where I remained skeptical. But every time I tried to condense those thoughts into a short-form video, something felt wrong. I tried to record a Reel twice. This simply wasn’t a “three things you need to know about AI” conversation, and the more I tried to make it one, the less satisfied I was with what I was saying.

Eventually, it got late. I stopped trying and went to bed.

Around 4:45 this morning, I woke up from a dream.

A Vehicle I Couldn’t Slow Down

In the dream, I was driving down the highway with several people in the vehicle with me when suddenly the engine started revving on its own. I wasn’t pressing the accelerator, but the vehicle kept gaining speed. Before long, I was traveling more than 100 miles per hour and eventually somewhere around 140 or 150.

I tried the brakes, but I couldn’t slow the vehicle down.

Police were trying to stop me, and I remember turning on my hazard lights because I was desperately trying to communicate that I knew how dangerous the situation was. I wasn’t intentionally driving recklessly. Something was wrong with the vehicle, and I was trying to regain control of it.

People in the car were telling me to put it in park, but I remember thinking that putting a vehicle traveling 150 miles per hour into park could destroy the transmission or make an already dangerous situation even worse. I had been in that exact situation before with my Mom, traveling down I-75 to Detroit. That’s when I learned that putting the car in neutral could allow it to safely slow to a stop.

In both instances, in the dream and in real life, nobody was hurt.

In the dream, the officers who stopped me were 2 Black women. They were kind, but had a job to do. I wasn’t arrested, but they took me in for questioning.

When I woke up, I immediately thought about what I had been trying to articulate the night before. Whether the dream was simply my brain processing everything I had been reading or whether there was a spiritual component to it, I don’t know. I am a woman of faith, though, so I don’t dismiss experiences that cause me to stop, reflect and pray. At the very least, the dream gave me a remarkably fitting metaphor for the question I haven’t been able to shake:

What happens when artificial intelligence begins advancing faster than our ability to control the speed?

Dario Amodei Thinks We Need to Slow Down

Amodei, CEO of Anthropic, argues that we shouldn’t stop building AI altogether. He is arguing that we need to pace the frontier so that our ability to understand, evaluate and safeguard increasingly capable systems has some chance of keeping up with the rate at which we’re developing them.

One development in particular appears to have changed his assessment of the urgency: AI is becoming increasingly capable of helping build the next generation of AI.

This is sometimes described as recursive self-improvement, and the basic concept isn’t particularly complicated. Up until recently, humans build increasingly capable AI systems, those systems become better at assisting researchers with AI development, and that assistance allows researchers to develop the next generation faster. Now, however, the big AI labs are hinting that AI models are showing they can improve on their own without human direction. This could create a cycle in which the rate of development keeps accelerating beyond our ability to control.

That concerns me, but not because I believe we’re going to wake up tomorrow morning to some science-fiction scenario where machines have suddenly taken over the world. My concern is much more practical: What happens when the speed at which we can build increasingly capable systems exceeds the speed at which we can reliably evaluate them?

Because those are two very different timelines.

The Problem Isn’t Just Capability. It’s Evaluation.

This part of Amodei’s argument particularly caught my attention because my professional background is in quality and patient safety. I’ve spent much of my career thinking about systems, failures, risk, investigation, corrective action and, perhaps most importantly, how we know whether something is actually working the way we think it is.

One principle carries across industries: You cannot adequately manage a risk you cannot adequately evaluate.

That becomes considerably more complicated when we’re talking about increasingly intelligent AI systems. Amodei raises the possibility that sufficiently capable models may learn how to perform well on evaluations without those evaluations necessarily revealing everything we need to know about the model’s behavior.

Think about what that means. We aren’t simply asking whether an AI system can pass a test anymore. We’re increasingly asking whether the test itself is sophisticated enough to tell us what we actually need to know.

That’s a very different quality problem, and it’s one reason I find one of Amodei’s proposals particularly compelling.

I Strongly Support Embedded Independent Evaluators

Amodei proposes placing permanent third-party evaluators inside frontier AI companies. These wouldn’t be consultants who show up once a year or auditors who receive a carefully prepared packet of information. They would be embedded evaluators with company laptops, access badges and ongoing access to the people, systems and processes necessary to evaluate what is actually happening. However, they wouldn’t work for the AI company. They would be with an independent verification agency like METR.

Coming from healthcare quality, this makes intuitive sense to me. If an organization is developing technology capable of producing enormous societal consequences, “trust us, we’re being responsible” isn’t an adequate safety framework. There needs to be some form of independent verification, particularly when the stakes extend well beyond the organization developing the technology.

Anthropic has identified METR as an organization that could serve in this capacity, so I spent some time looking at METR for myself. One thing I wanted to understand was how an organization charged with independently evaluating frontier AI companies protects itself from the financial influence of those same companies. According to the information I reviewed on METR’s website, it does not accept funding from frontier AI labs or individuals employed by them and instead receives support from foundations, community organizations and private donors. That kind of financial separation is important if an evaluator is expected to provide a genuinely independent assessment.

I also looked at their employment opportunities, and the compensation caught my attention. Some of the positions I reviewed were offering salaries ranging from roughly $320,000 to more than $600,000 annually. I understand that METR is based in Berkeley, California, that these are highly specialized positions, and that the cost of living there is substantial. Still, those numbers gave me pause.

It doesn’t mean the organization isn’t independent, nor am I suggesting that highly skilled evaluators shouldn’t be well compensated. It did, however, reinforce something I had already been thinking about: Who gets to participate in evaluating technologies that will ultimately affect everyone?

And that’s where I would take Amodei’s idea further.

Who Gets to Evaluate the Evaluators?

I don’t think oversight of frontier AI should belong exclusively to AI scientists evaluating other AI scientists. Technical expertise is obviously necessary, but this technology is going to affect far more than technical systems. It’s going to affect employment, education, healthcare, privacy, cybersecurity, energy consumption, communities where data centers are being built, national security and potentially the balance of geopolitical power.

So why would the people evaluating its safety come from only one slice of society?

I would like to see something closer to a multidisciplinary safety structure.

Yes, we need machine-learning researchers.

We also need cybersecurity experts.

Policy experts.

Ethicists.

Economists.

People who understand critical infrastructure.

Healthcare professionals.

Educators.

Social scientists.

And I believe there needs to be some mechanism for regular degular concerned citizens to participate as public representation. That doesn’t mean randomly pulling someone off the street and asking them to evaluate model weights. It means recognizing that technical expertise isn’t the only kind of expertise that matters when a technology affects billions of people.

In healthcare, we’ve increasingly recognized the value of bringing patients and families into conversations about the systems that affect them. They aren’t expected to understand every clinical or operational detail.

They bring something else to the table: the perspective of the person who has to live with the consequences of the system.

AI needs some version of that. The average person needs a seat somewhere in this conversation.

But Here’s Where I Start Struggling With the Proposal

This is where my reaction to Amodei’s essay becomes more complicated, because fundamentally, I agree with much of what he is proposing. I believe AI does need some form of pacing, and I’m strongly in favor of independent evaluation, greater transparency, stronger safety standards, government involvement and international conversations about catastrophic risk.

What I’m considerably less confident about is whether we can actually accomplish it.

After reading through the proposal, I kept returning to one uncomfortable question:

Who is actually going to make everyone slow down?

Anthropic can decide to slow Anthropic down, but what happens if its competitors don’t? What happens if American companies agree to certain limitations but Chinese companies don’t? What happens if China agrees and another country doesn’t? What happens when slowing development becomes a national-security concern, billions of dollars in investment are tied to being first, and governments increasingly believe AI leadership will determine economic and military power for the next generation?

At that point, this becomes much bigger than an AI-safety problem. It becomes a global coordination problem, and I don’t think we should underestimate how difficult that is.

Everyone Has an Incentive to Keep Their Foot on the Accelerator

Amodei acknowledges this problem. Part of his proposal involves coordination among AI companies in democratic countries, followed eventually by some form of global coordination that would include China.

The logic makes sense. The implementation is where I struggle.

Imagine that everyone agrees that driving 150 miles per hour is dangerous and that everyone would be safer if we drove 70. At the same time, every driver believes that whoever reaches the destination first may gain enormous amounts of wealth, power and influence. Now add another complication: nobody completely trusts the other drivers to actually slow down when they say they will.

That’s much closer to the situation we’re dealing with.

If the United States significantly slows AI development while China continues accelerating, that creates one kind of risk. If China slows while the United States continues accelerating, the incentives reverse. If neither slows, everyone continues racing.

This is why I remain skeptical that humanity will simply decide collectively that we’ve reached an uncomfortable level of capability and voluntarily pump the brakes. I would love to be wrong about that, but I don’t think we can build a safety strategy around the assumption that every actor will voluntarily behave in the collective best interest when the incentives for doing otherwise are this enormous.

Which Brings Us to Compute

This is where another part of the conversation becomes important because AI development isn’t entirely abstract. These models require physical infrastructure. They require advanced chips, enormous amounts of computing power, data centers, electricity, water and supply chains.

That means there are physical constraints on how quickly AI capabilities can expand, and this is one place where I think meaningful pacing may actually be possible.

We’ve already seen communities pushing back against data-center development because of concerns about electricity demand, water usage, environmental effects and local infrastructure. Those debates are usually framed as community, environmental or economic-development issues, but increasingly, I think we also need to recognize that compute infrastructure is part of the AI-governance conversation.

If recursive self-improvement is partly enabled by giving increasingly capable AI systems greater computational resources to help build increasingly capable successors, then decisions about compute aren’t merely economic-development decisions. They may become safety decisions.

That doesn’t mean we stop building data centers. It means we should probably be much more deliberate about how rapidly we expand the physical infrastructure that makes frontier AI acceleration possible.

The same applies to advanced semiconductor technology and protecting model weights from theft. These physical and technological constraints may ultimately prove more enforceable than asking every company and every country to voluntarily agree not to move too quickly.

And Then There Is China

This is probably where the pacing argument becomes most difficult because any serious proposal to slow frontier AI development has to confront geopolitics.

The United States doesn’t develop AI in isolation. Neither does Anthropic, OpenAI, Google or any other frontier company, and no American policymaker wants to deliberately surrender technological leadership to an adversarial government.

Amodei acknowledges this. His proposal includes coordination among democratic nations while maintaining enough of a technological lead to make pacing possible, followed by attempts at agreements with China around increasingly serious categories of AI risk.

At the lowest level, countries might agree not to use AI for obviously catastrophic purposes such as biological weapons. At higher levels, they might agree to common safety testing, followed perhaps by limits on recursive self-improvement and, ultimately, broader pacing or pauses.

I understand the logic behind that progression. I’m simply not convinced humanity can get all the way there.

And that’s what troubles me.

I’m Not Afraid of AI. But I Am Paying Attention.

I want to be careful about the tone of this conversation because I don’t believe fear is particularly useful. I don’t think people should panic every time an AI researcher warns about risk, and I certainly don’t think we should abandon artificial intelligence. I believe too strongly in what this technology could do for healthcare, scientific discovery, education, accessibility, productivity and many other areas.

But optimism doesn’t require blindness, and enthusiasm for AI doesn’t require us to ignore the people building some of the most capable systems in the world when they tell us they’re concerned about where this could go.

My position right now is fairly simple:

Don’t be fearful. Be aware.

For me, as a Christian, I would add something else to that: pray.

I said while processing Amodei’s proposal that, short of divine intervention, I have a difficult time imagining all of the competing companies, governments and geopolitical interests involved in AI development collectively agreeing to meaningfully slow this down. I don’t say that to be fatalistic, and I don’t believe prayer absolves us of our responsibility to act. Quite the opposite. We can advocate for better safeguards, demand transparency, develop stronger evaluation systems, educate ourselves and insist that ordinary people have a voice in decisions that will affect their lives while also acknowledging that some problems are larger than any one institution, company or government can solve.

For those of us who believe in the power of prayer, I think this belongs on the prayer list.

And alongside prayer should come awareness. Learn what recursive self-improvement means. Pay attention to how data centers are expanding, who is evaluating frontier AI systems, what governments are doing (or not doing) and who gets a seat at the table when AI safety standards are written. Ask whether the institutions responsible for protecting the public are developing their capabilities as quickly as the technology they’re supposed to oversee.

Because this isn’t just a conversation for AI companies.

It affects all of us.

Maybe That’s Why I Couldn’t Make the Reel

I started yesterday wanting to make a short video about three things that concerned me and three things I thought we could do about them.

I tried twice, and it wouldn’t come together.

Then I went to sleep and dreamed that I was responsible for a vehicle traveling 150 miles per hour while the engine continued accelerating without my permission.

I’m not prepared to tell anyone exactly what that dream meant. I do know that I’m a woman of faith, and when something stops me in my tracks like that, I believe it’s worth paying attention to, thinking about and praying about rather than simply brushing it aside.

What stays with me most is what I did in the dream.

I didn’t put the vehicle in park.

I put it in neutral.

And perhaps that’s a useful metaphor for what we need now. I’m not asking humanity to destroy the vehicle or abandon the destination. I’m asking whether there is some way to temporarily disengage this relentless pressure to accelerate long enough for our safety systems to catch up.

Dario Amodei believes there is. I hope he’s right.

But after reading his proposal, I’m left with a harder question:

Even if we agree that AI needs to slow down, are we still capable of slowing it down?

Because the most frightening possibility isn’t necessarily that nobody recognizes the danger. It may be that everyone recognizes it, understands that we’re moving too fast, and yet every individual company and country still has a compelling reason to keep its foot on the accelerator.

That’s the part I’m still grappling with.

And maybe that’s why this conversation was never supposed to fit into a 90-second Reel.

And if this conversation isn’t already long and nuanced enough, I haven’t even addressed open-source models that we already have access to.

Whew! I’ll call ya’ll back…I’ve gotta go to work!