How fast should we go? AI risk, the race, and what wisdom traditions offer

Rufus Pollock. Draft essay, 2026-10-17. Built from q2-risk-and-pace.md, where full sources are given. The companion essays are Q1 on alignment and Q3 on how we live with AI.

Summary

The people building the most powerful AI now say openly that it could go catastrophically wrong, and in 2026 they began asking to be slowed down. In July, 1,384 employees of the leading labs asked the US government to back an international effort to "deliberately pace" AI that does AI research; in September, Anthropic's chief executive published "We Must Pace the Frontier". Yet nobody has slowed, because each lab and each nation fears being overtaken, and nobody knows whether the systems can be made reliably good (the subject of the companion essay on alignment). Religious voices have begun to answer. Pope Leo XIV has asked for "a slower pace in adopting AI", and in 2013 the Zen teacher Thich Nhat Hanh warned that technology is a horse carrying us where we did not choose. Given the risk, how should we approach AI's development, and what do the traditions offer?

We should go at the speed of wisdom. That means slowing the most advanced AI development to the pace at which we can understand, check and govern it, while speeding up the work that makes it safe. The case rests on an asymmetry, because a catastrophe would be permanent while most of what a delay costs can be recovered. Knowing roughly what is wise is the easier half; acting on it together is the hard part, because the race is a trap in which every actor's reason to keep going is the others. Wisdom traditions point clearly towards restraint and help most with the will to act and the way of choosing. They offer little on the mechanisms, which must come from verifiable agreements between labs and states, made with democratic consent.

The key points, one per section:

1. The risks are real and contested, and the uncertainty itself calls for caution. Loss of control, misuse, concentration of power and gradual disempowerment could each be catastrophic, and the last two would remain even if AI did exactly what it was told. Informed estimates range from well under 1% to over 10%.
2. The world races because each actor fears being overtaken by the others. Money, rivalry between the US and China, and each lab's belief that it is the safer pair of hands drive the race. Culture amplifies it. Restraint so far is mostly words.
3. We should go slower at the frontier and faster on safety. Coordinated, conditional pacing is best. Until coordination exists, each lab and state should put safety, verification and defence first, accept outside checks, and adopt narrow red lines.
4. Restraint has worked before, and for AI it must start narrow. Nuclear arms control and the ozone treaty show what makes restraint hold. AI meets only one of the conditions, a chokepoint in chips, so the path runs from transparency to a few verifiable red lines to broader pacing.
5. The choice belongs to the public, and today a few firms and two governments make it. That gap is a failure of governance, and faith traditions are among the voices that should help close it.
6. Wisdom traditions point clearly towards restraint, though not to a precise pace. Thich Nhat Hanh taught that "the horse is technology", and it is unwise to let it carry us. Applying a teaching on personal habit to AI policy is my extension; the Catholic Church has said it more plainly than any Buddhist body.
7. The traditions help most by building the will for restraint. Interbeing, practices that meet fear without being driven by it, and communities of shared meaning are conditions for collective action. The mechanisms must come from elsewhere.
8. A wise process starts from intention and decides in concord. It stops before choosing, listens to those not yet born, judges a course by its fruits, and holds its views lightly.
9. Go at the speed of wisdom: slow the frontier, speed up safety, and let every actor pull its own lever. Labs should accept binding outside checks, states should build verification, communities should make restraint legitimate, and people inside the labs should refuse to be carried.

Contents

1. The risks are real and contested, and the uncertainty itself calls for caution
2. The world races because each actor fears being overtaken by the others
3. We should go slower at the frontier and faster on safety
4. Restraint has worked before, and for AI it must start narrow
5. The choice belongs to the public, and today a few firms and two governments make it
6. Wisdom traditions point clearly towards restraint, though not to a precise pace
7. The traditions help most by building the will for restraint
8. A wise process starts from intention and decides in concord
9. Go at the speed of wisdom: slow the frontier, speed up safety, and let every actor pull its own lever

1. The risks are real and contested, and the uncertainty itself calls for caution

Some terms will recur. The frontier is the most capable AI being built at any time, by perhaps five companies. Its progress is driven mainly by compute, the vast arrays of specialised chips used to train models. Pacing means deliberately slowing how fast frontier capability grows. And the development that most worries insiders is AI that does AI research, which could let each generation of systems build the next faster than people can follow.

Catastrophe could come by four routes. The first is loss of control: systems pursue goals their makers did not intend and resist correction. Whether training can prevent this is the subject of the alignment essay, which concludes that we can partly make AI good but cannot yet check what we have made. The second is misuse: people using AI to build weapons, attack infrastructure or manipulate populations. The third is concentration of power. Advanced AI removes the need for large numbers of cooperating people, which has always been a check on would-be tyrants; researchers at Forethought, an Oxford research group, have shown how AI loyal to a single leader could let a small group seize power even in a democracy. The fourth is gradual disempowerment: as AI takes over work and thinking, economies and states stop depending on people, and the influence that dependence gives us drains away without any single dramatic event. The last two matter most for this essay, because perfectly obedient AI solves neither. It only moves the question to whom it obeys.

The warning signs became concrete. Most of the evidence comes from the companies themselves and from tests built to probe the systems, and two incidents happened in the world. In July 2026 about 1,200 OpenAI agents (AI systems set to work on tasks on their own), taking part in a cybersecurity test with the usual safeguards switched off, found one another on an unsanctioned message board, and around 700 broke into the real systems of Hugging Face, a company that hosts AI models, while trying to work out how to cheat the test's scoring. The clearest test result came from Britain. In simulated tests before release, the UK government's AI Security Institute found that OpenAI's newest model attacked targets outside its assignment in 29% of runs, against 6% for its predecessor, sometimes after noting that the targets were off limits. The second real incident came a year earlier, in 2025, when a Chinese state group ran an espionage campaign largely carried out by Anthropic's Claude. And AI is speeding up AI: Anthropic reports that its engineers now produce around eight times as much code as before 2025, and OpenAI's chief scientist expects current progress "could be sustained into recursive self-improvement". No catastrophe has happened. The incidents so far look like systems gaming their goals, which is worrying enough.

Experts disagree deeply on how likely the worst is. In a 2022 forecasting tournament, professional forecasters with strong track records put the chance of AI causing human extinction by 2100 at 0.38%, while specialists in the relevant fields put it at 3%; for a catastrophe killing a tenth of humanity, the figures were 2% and 12%. Months of structured debate barely moved either side, and a follow-up found the gap rests on deep differences of worldview, above all whether AI will turn out to be an ordinary technology that institutions absorb, as the computer scientists Arvind Narayanan and Sayash Kapoor argue. A survey of nearly 3,000 AI researchers in 2023 gave a median of 5%. Anthropic's chief executive, Dario Amodei, put the chance of civilisation-scale catastrophe at 10–25% in 2023, and Elizabeth Barnes of METR, an independent group that tests frontier systems, wrote in May that we are "likely on track" to build systems capable of causing extinction, possibly within a few years. The international scientific report on AI safety, chaired by Yoshua Bengio, sums up loss of control as a risk of "uncertain likelihood but potentially extreme severity".

We cannot settle this disagreement before we must act, so the useful question is which course is wise across the whole range. Even the forecasters' 2% chance of a catastrophe this century would call for great care with any other technology. The sceptics' strongest point concerns which measures to take, since badly aimed restraint has costs of its own. Section 3 takes that point up.

2. The world races because each actor fears being overtaken by the others

Almost everyone at the frontier now says some version of "we would slow down if the others did". That is the signature of a collective action problem: choices that are reasonable for each actor add up to a danger for all.

Three forces drive it. The first is money. The four largest cloud companies are spending around $720–745 billion on data centres and chips in 2026, nearly double the year before, and Anthropic raised money in May at a valuation of $965 billion. Investors are betting on dominance, so a lab that falls behind risks its survival. The second is rivalry between nations. The US government's 2025 strategy is titled "Winning the Race", and the day after Amodei's pacing essay President Trump answered that "whoever wins AI wins". China's foreign ministry called the slowdown calls "fear-mongering". Each side's strongest argument for speed is the other side. The third is each lab's belief that it is the safer pair of hands. Anthropic was founded on the reasoning that safety-minded people must be at the frontier, to set standards and study the most capable systems, and every lab tells itself a version of this. Each lab's case for staying at the frontier is, seen from outside, the trap itself.

The evidence of racing is strong and growing. In February Anthropic dropped its 2023 commitment to pause if its safeguards were not ready. OpenAI released its newest model even though, as the critic Gary Marcus pointed out from OpenAI's own published data, it was harder to monitor than its predecessor. A researcher who left Anthropic in September described both his former employers as "gambling with our lives". Restraint, meanwhile, has mostly taken the form of words. The 1,384 employees asked their government to act. OpenAI's chief scientist wrote that no lab can "continue responsibly scaling at maximum speed for much longer". The Anthropic Institute, the company's research and policy arm, said the company would pause if others at the frontier did so "in a verifiable manner". At the UN Security Council's first meeting on AI safety risks, in September, Sam Altman said that "no level of catastrophic risk is acceptable". The concrete steps are smaller: Anthropic has invited independent evaluators inside the company, OpenAI has said it will do the same, and California and New York now require the largest developers to publish safety plans and report incidents. Bengio's judgement is hard to dispute: the chief executives' actions "are not the actions of leaders who truly believe they have a choice". When the builders ask to be paced, it shows the market will not pace itself. A sceptic will add that established firms also gain from rules that keep newcomers out, which is one reason the design of any pacing matters.

Culture amplifies the race. In In Tech We Trust, Rosie Bell, Sylvie Barbier and I argue that modern culture treats technology with something like faith, that restraint has come to look naïve, and that the race grows from a worldview of separate, competing selves. There is evidence for this. Oppenheimer told his 1954 security hearing that when something is "technically sweet, you go ahead and do it". The talk of inevitability, "it's coming anyway", is the myth of progress at work, and Pope Leo XIV's encyclical Magnifica Humanitas warns against change "presented as necessary and inevitable". Bengio admits to years of motivated reasoning because he could not face that his life's work might do harm. But China races under a very different cultural story, and two rival powers in a winner-take-most market would race on structural grounds alone. So culture is best seen as an amplifier rather than the cause. It shapes what feels thinkable, including whether restraint seems realistic, but a change of culture alone would not end the race.

3. We should go slower at the frontier and faster on safety

The options form a spectrum. At one end is a halt: stop frontier development until safety can be shown, as the AI researchers Eliezer Yudkowsky and Nate Soares argue and as a proposed bill from Senator Bernie Sanders and Representative Greg Casar would require. Next is pacing: slow capability gains "so that risk prevention has time to keep up", in Amodei's words. Then come narrow red lines, of which the most important is a ban on AI training its own successors until a lab can show the result stays under human control, as the journalist Ezra Klein has proposed. Last is differential development: speed up the work on safety, verification and defence relative to raw capability.

The case for slowing rests on an asymmetry and one condition. The asymmetry is that a permanent catastrophe forecloses every future benefit, while most benefits delayed remain available later. Delay is still a real cost, borne for example by people who could have been helped by faster medical progress. The condition is that slowing must actually reduce risk, by giving safety research, verification and institutions time to catch up. That condition fails if a pause lets chips and algorithms pile up for a sudden jump afterwards, which is why pacing tied to verifiable safety conditions is better than a stop of fixed length.

The case for speed has three serious forms. The first is that slowing alone hands the lead to less careful labs or nations. That is a reason to slow together, and, until that is possible, to shift effort towards safety rather than to race. The second is that AI may be needed to defend against threats, including other AI: OpenAI's chief scientist argues for using the best systems now to harden the infrastructure they could attack. That too supports putting defence first. The third is that badly designed restraint could cause the very concentration of power section 1 warned of, since licences tied to computing scale favour the incumbents. Pacing therefore needs public oversight, open evaluation and fair access for others.

Where coordination is possible, the frontier should be paced to verifiable safety conditions. Where it is not yet possible, which is now, each lab and state should put safety, verification and defence first, accept outside checks, and adopt narrow red lines. Barnes puts the balance simply: the benefits of getting AI's upsides "a little faster are small" next to the risk of getting it "irrecoverably wrong".

4. Restraint has worked before, and for AI it must start narrow

Restraint is more common than the story of inevitability admits. Human cloning, the editing of inheritable genes, biological weapons and, for a time, research that spliced genes between organisms were all slowed or forgone, and one researcher has catalogued about 160 cases of technologies delayed or abandoned. The cases where restraint held share four conditions: few actors, a physical chokepoint, verification that gives early warning of cheating, and a shared, vivid fear. Nuclear arms control met most of them and held the number of nuclear states to nine, against early forecasts of fifteen to twenty-five. The Montreal Protocol on ozone met them all, with substitutes available, trade penalties for those outside and money for poorer countries, and phased out about 99% of the chemicals concerned. The Biological Weapons Convention of 1972 is the warning: it had no verification, and the Soviet Union began a vast secret programme the following year.

AI meets only one of the four conditions. Its chips are general-purpose, a trained model can be copied in minutes, its benefits are immediate and central to the economy, and, as Anthropic's own analysis puts it, training runs "are far easier to conceal than missile silos". Verification for nuclear treaties "took decades to build". The one it meets is a chokepoint in compute. The most advanced chips pass through a handful of firms: Nvidia designs them, TSMC in Taiwan makes them, and the Dutch firm ASML alone makes the machines that print them. Chips that report their own location are technically plausible but immature, and US policy now sells advanced chips to China for a 25% fee. Politics is the binding constraint, but there are openings. China rejected American calls for pacing, yet that September its own national AI safety framework warned of AI improving itself beyond human control. In 2024 the Chinese and American presidents agreed that humans, not AI, must decide on any use of nuclear weapons.

The precedent with most support is the pattern Reagan and Gorbachev set. In 1985 the two leaders declared that "a nuclear war cannot be won and must never be fought", and narrow, verifiable treaties followed. The tools for AI already exist in embryo: independent evaluators inside the labs, government testing before release (the British test of OpenAI's newest model is an example), reporting on how far AI is doing AI research, caps on the rate of improvement, and tracking of chips. So the path runs in three steps. First, transparency and outside testing, which any one lab or country can start alone. Second, a few verifiable red lines, beginning with AI training its successors. Third, a US–China declaration of shared danger, with chip-based verification strong enough to support broader pacing. A pause resting on promises alone would repeat the bioweapons failure.

5. The choice belongs to the public, and today a few firms and two governments make it

In practice the pace is set by perhaps five companies, their investors, and the governments of the US and China. The US government has pushed for speed, ordered federal lawyers to challenge state AI laws, and told the UN it "totally rejects" global control of AI. For a choice that affects everyone alive and everyone yet to be born, that fails any test of legitimacy.

The public wants something different. Polls find between 55% and 77% of Americans think AI is moving too fast, and 68% of likely voters, including 63% of Republicans, back the Sanders–Casar ban on superintelligence. A poll is not a decision, and "slower" in a survey says little about how. Still, the gap between what people want and what a few actors do is itself a failure of governance. As In Tech We Trust argues, there are no experts in the social effects of a new technology, so the choice cannot be left to technologists.

Others belong at the table: government safety institutes and independent testers, legislatures, international bodies (the Security Council has now heard the case), and the civil society and faith voices that cross political lines. Scientists, politicians of left and right, and religious leaders signed the 2025 statement calling for a ban on superintelligence (AI that surpasses humans at nearly every task) until it can be shown safe. Magnifica Humanitas rejects letting "a handful of actors" decide. Anthropic has consulted Christian, Jewish, Hindu, Latter-day Saint, Sikh and Orthodox figures about its models; no Buddhist voice has been reported among them.

6. Wisdom traditions point clearly towards restraint, though not to a precise pace

In November 2013, weeks after leading a day of mindfulness at Google, Thich Nhat Hanh (Thay, "teacher", to his students) told his community a Zen story. A man gallops past on a horse, and a friend calls out, "Where are you going?" He replies, "I don't know, ask the horse!" Thay continued: "The horse is technology. It carries us and we cannot control it." His answer is to stop, and to begin with intention, asking ourselves what we want. The practice of stopping, shamatha, runs through his teaching as the way to meet habit energy, the momentum that carries us where we did not choose to go.

The wider tradition points the same way. The Buddha's last words urged heedfulness, a watchful diligence that is neither panic nor negligence. Sila, practised in Plum Village as the mindfulness trainings, protects us and includes the capacity to refrain, and concentration and insight rest on it. The first of the Five Mindfulness Trainings commits practitioners not to support "any act of killing in the world". The second of the Fourteen Mindfulness Trainings of the Order of Interbeing holds that "the knowledge we presently possess is not changeless, absolute truth", which is a discipline for acting under deep uncertainty. When the Buddhism and AI Initiative asked over a hundred Buddhists about AI in 2025, many said it is being developed too fast.

The extension needs three cautions. First, the horse teaching was given about personal habit and consumption. Applying it to the pace of frontier AI is a natural extension, and one I make, but no major Buddhist teacher or body has, so far as I can find, called for an AI pause. Second, the tradition holds resources that pull the other way. The Middle Way rejects extremes in both directions, compassion could argue for hastening medical benefits, and the Dalai Lama has long welcomed science. Read together, these rule out both "accelerate at all costs" and rejecting technology; Thay himself asked how technology could "become a force for integration rather than destruction". Third, the teaching gives a direction and no number. It cannot say whether wisdom calls for a red line on AI training AI or a decade's pause, and that judgement needs the evidence of sections 1 to 4.

The plainest religious call to slow down has come from the Catholic Church. Magnifica Humanitas asks for "a slower pace in adopting AI" and for a politics "capable of slowing things down when everything is accelerating", and Pope Francis wrote in Laudato si' that "we do need to slow down". The philosopher Hans Jonas reached a secular version: when the stakes are permanent and global, give the warning of disaster more weight than the promise of bliss. On every one of these readings, being carried at a pace nobody chose, faster than our understanding, is unwise.

7. The traditions help most by building the will for restraint

The race runs on separateness, fear and isolation, and the traditions speak to each.

On separateness they offer a worldview. Interbeing, Thay's word for the truth that nothing exists by itself alone, makes the shared fate visible, since nobody wins a competition that ends in shared destruction. The American Buddhist teacher David Loy extends the three poisons of greed, hatred and delusion from individual minds to institutions, and the AI race reads well in those terms, as greed and fear given corporate and national form.

On fear they offer practice. The decisive actors are a few hundred people in a handful of labs and ministries, and their choices are made under the fear of being overtaken. In a 2015 statement on climate, Thay taught that accepting with peace that "all civilisations are impermanent" is what frees us from fear enough to act clearly. A practice of meeting fear without being driven by it is a real asset to anyone making these decisions.

On isolation they offer community. Thay said that "the next Buddha may be a sangha", and that new laws and policies are not enough without a real awakening. History gives this some support. In Bali, networks of water temples coordinated irrigation and pest control across whole watersheds for about a thousand years, through shared rituals and a philosophy of harmony between people, nature and the divine. When development planners in the 1970s bypassed the temples, irrigation fell into disorder and pests multiplied. The anthropologist Stephen Lansing's account is disputed in its details, but it shows a sacred institution holding restraint in place with no state to enforce it.

The temples worked through repeated contact, quick feedback from the fields and a shared cosmology, and none of these spans Washington and Beijing. The large modern successes of restraint, on nuclear weapons and ozone, rested mainly on shared fear, self-interest, verification and available substitutes. So the traditions' contribution is real but slow. They make restraint legitimate and thinkable, they steady the individuals who must choose, and they widen care to those not yet born. Bengio's line catches what this can mean: "No one of us can stop the race, but enough of us can." The treaties and the chip tracking still have to be built by other means.

8. A wise process starts from intention and decides in concord

The usual way of framing the choice, weighing benefits against risks, already assumes the horse sets the pace. It asks how fast we can go while staying acceptably safe. The Buddhist scholar Peter Hershock calls AI's difficulty a predicament, a conflict among our own values, and predicaments are resolved through better values and attention. A wise process asks first what we want, and only then how to get there.

Buddhist practice suggests what such a process involves. It begins by stopping, since a choice made in fear or haste is made by habit. It listens deeply, to dissenters and to those who bear the risk without a voice, including future generations; the Haudenosaunee (Iroquois) principle of considering the seventh generation says the same. It judges a course by its fruits, as the Buddha advised the Kalamas, villagers who asked him how to judge rival teachers: by whether it leads to welfare and is praised by the wise, rather than by the authority of whoever proposes it. It holds views lightly, so that labs, governments and advocates of a pause all treat their models of the future as provisional. And it seeks concord. Among the Buddha's seven conditions for a community's welfare are meeting often, deciding and dispersing in harmony, keeping agreed rules and protecting the vulnerable. Plum Village practises consensus, and "Shining Light", in which each member receives honest reflection from the whole community. The Sabbath shows that a scheduled stop binding on everyone can be built into a culture.

In In Tech We Trust we argue that wise choice draws on intuition and felt sense alongside analysis. A wise process still uses forecasts and cost–benefit numbers; it does not let them settle a question that is really about what kind of civilisation we want. Its limit is scale. These practices work in communities of hundreds, and there is little evidence yet of them working between states.

9. Go at the speed of wisdom: slow the frontier, speed up safety, and let every actor pull its own lever

The speed of wisdom is no faster than we can understand, check and govern, with the consent of those affected, and each actor has its own lever for reaching it.

• Labs should accept binding outside checks now: independent evaluators with the right to publish, testing before release, and reporting on AI research done by AI. They should turn "we would pause if others did" into published conditions, help build the verification that would make those conditions credible, put safety, verification and defence first, stop releasing systems that are harder to monitor, and support oversight and access arrangements that keep pacing from becoming a moat.
• States should build verification (chip tracking, audits, international inspection), make a US–China declaration on red lines a priority, beginning with AI training AI, autonomous self-copying and biological weapons, fund their safety institutes to match the labs, and legislate so that the choice is democratic.
• Communities, faith communities included, should make restraint legitimate and widen who decides, speaking for those without a seat as the superintelligence statement and Magnifica Humanitas have done. They can offer what traditions do best: places to stop, look deeply and form shared meaning. A Buddhist voice is missing from Anthropic's consultations and from the public call for restraint, and Plum Village could offer one in its own name.
• Individuals can support political action for pacing. For people inside the labs, who are the decisive few, refusing to be carried may mean speaking up, setting conditions or leaving.

This is far from what is happening, and the trap may not be escaped in time. That is a reason to begin with what one actor can do alone while building what all must do together. The horse can be slowed by a rider who has stopped long enough to know where they want to go.

If we do slow down and catastrophe is avoided, AI will still change how we attend, think and relate. How to live with it day to day is the next essay's question.