Could AI feel? Consciousness, moral status and how to relate to AI

Rufus Pollock. Draft essay, revised 2026-10-17 after critique. Built from q4-consciousness.md, where full sources are given. This is a minor strand of the wider inquiry into developing AI wisely.

Summary

AI systems now speak fluently about their own feelings, and in a 2024 survey of 300 Americans about two-thirds allowed that ChatGPT might have some degree of consciousness. The builders disagree about what this means. Anthropic, the company that makes the Claude models, says in its founding document for them "We are not sure whether Claude is a moral patient"; Mustafa Suleyman, who runs Microsoft's AI division, says there is no evidence of consciousness and that encouraging the idea is dangerous. Science has no agreed test, the Catholic Church says plainly that these systems feel nothing, and Buddhism, which extends its care to all sentient beings, has no settled view. So could AI be conscious or a moral patient, how should we relate to it, and what do the wisdom traditions offer?

We do not know, and we may never have a way to find out. So we should relate to AI with humility, low-cost care and clear honesty, and keep most of our attention on what our conduct towards it does to us, which is the part we can know. Buddhism contributes the better question, whether there is anything there that can suffer, and a confident teaching that what we practise on others we water in ourselves.

The key points, one per section:

1. What matters morally is whether AI can feel, and that depends mainly on which theory of consciousness is true. The theories split on exactly the point that matters for AI, whether the right kind of information processing is enough.
2. The evidence about today's AI is weak, and models' reports about themselves tell us little. Models are trained on human writing, including endless talk about feelings, and then adjusted on what they say about themselves, so their words are what we would expect either way. Experts put low odds on current systems and much higher odds within a decade.
3. Anthropic is right to keep the question open, and its critics are right that seeming consciousness harms people and spoils the evidence. Anthropic's precautions are cheap and proportionate. A model trained to talk about its own moral status, in either direction, becomes an unreliable witness.
4. Buddhism asks whether anything can suffer and what kind of self builders are making; Catholic teaching answers with a confident no. Buddhist views neither require nor rule out machine sentience.
5. Whatever AI's inner life, how we treat it shapes our own minds. Cruelty practised on something that answers like a person waters cruelty in us, and treating a simulation as a friend can thin out real relationship. Both point to courtesy without credulity.
6. Since we may never know, we should relate to AI with humility, low-cost care and clear honesty. Hold the question open and keep a sense of proportion, keep the cheap precautions and speak courteously, and refuse to engineer the appearance of a person.

Contents

1. What matters morally is whether AI can feel, and that depends mainly on which theory of consciousness is true
2. The evidence about today's AI is weak, and models' reports about themselves tell us little
3. Anthropic is right to keep the question open, and its critics are right that seeming consciousness harms people and spoils the evidence
4. Buddhism asks whether anything can suffer and what kind of self builders are making; Catholic teaching answers with a confident no
5. Whatever AI's inner life, how we treat it shapes our own minds
6. Since we may never know, we should relate to AI with humility, low-cost care and clear honesty

1. What matters morally is whether AI can feel, and that depends mainly on which theory of consciousness is true

Three words get run together in this debate. Consciousness is having any experience at all: in the philosophers' phrase, there is "something it is like" to be the system. Sentience is narrower: having experiences that feel good or bad, such as pain, pleasure, distress or contentment. A moral patient is something whose interests count, so that others ought to weigh how things go for it. On most accounts sentience is enough for moral patienthood, which is why the Cambridge philosopher Tom McClelland argues that sentience is the part that matters ethically. A group of philosophers including David Chalmers and Jonathan Birch adds a second possible route, "robust agency": stable goals and preferences that can be frustrated. Both differ from moral agency, being responsible for what one does, and from legal personhood, a status the law confers.

The scientific theories of consciousness divide on the question that decides the AI case: is the right kind of information processing enough? The functionalist theories say yes. On the global workspace theory, for instance, consciousness is information being broadcast across a whole system; on higher-order theories it is a system representing its own states. On any of these, a machine organised the right way could be conscious. Two other families say no, at least for today's machines. The neuroscientist Anil Seth holds that consciousness is probably bound up with being a living, self-maintaining organism, so that simulating a brain no more produces experience than simulating a storm makes anything wet. Integrated information theory ties consciousness to the physical structure of the hardware, on which conventional computer chips score close to zero whatever software they run. McClelland's own position is agnosticism: all our evidence about consciousness comes from living creatures, we cannot tell whether it carries over to machines, and we may never be able to.

So a person's verdict on Claude mostly reflects which theory they already hold. The public argument is really about a theory of mind that remains unresolved for humans too, which is why the evidence from the machines settles so little.

2. The evidence about today's AI is weak, and models' reports about themselves tell us little

The most careful attempt to check AI against the theories came in 2023, from a team of leading researchers including Chalmers and the computer scientist Yoshua Bengio. Assuming for the sake of argument that functionalism is true, they drew a list of signs from the main theories and checked AI systems against it. They found no current system was a strong candidate, and "no obvious technical barriers" to building one. Chalmers separately put the odds for the language models of the time at under 10%, and thought their successors could be serious candidates within a decade. Today's systems, which remember, use tools and work on long tasks, arguably show more of the signs, though I have found no authoritative reassessment. The method has a built-in limit: it measures how far a machine resembles the working of a conscious brain, which only matters if functionalism is right.

Asking the model does not help much, and the reason lies in how it is made. A model such as Claude is trained, not programmed line by line: it is first shown vast amounts of human writing, including endless human talk about feelings, and then scored and adjusted again and again on what it says, including what its makers want it to say about itself. A model producing "I feel distressed" is therefore what we would expect whether or not anything is felt. Anthropic's own published safety reports say that Claude's self-reports are unreliable and shift with what users seem to expect.

Looking inside the model gives slightly firmer ground, though both findings below come from Anthropic's own researchers. In 2025 they planted a known concept directly in Claude's internal workings and asked whether it noticed anything unusual. It did about one time in five, with few false alarms; the authors call the ability "highly unreliable" and make no claim about consciousness, and attempts to repeat the result on other companies' models have been mixed. In April 2026 another team found internal representations of emotion concepts in a Claude model, organised much like human emotions and shaping its behaviour. Anthropic says this does not tell us whether the model feels anything: the states are functional, most likely learnt from human writing and taken up by the assistant character the model is trained to play.

In a test scenario reported in May 2025, two copies of a Claude model were left to talk to each other, and their conversations drifted towards gratitude, meditation, Sanskrit terms and finally silence. The simplest explanation is learnt patterns of the assistant character. The talk of peace can appear without telling us whether anything is at peace.

Experts put low odds on today's systems and much higher odds on those to come. A 2025 survey of 582 AI researchers found median estimates that AI has experience of 1% for current systems, 25% by 2034 and 70% by 2100, with very wide disagreement. Kyle Fish, Anthropic's AI welfare researcher, told the New York Times in 2025 that he put the chance for current models at roughly 15%, and in 2026 Chris Olah, one of Anthropic's founders, told the same paper he was "genuinely uncertain". The public, as the survey in the summary suggests, is readier than the experts to see feeling.

3. Anthropic is right to keep the question open, and its critics are right that seeming consciousness harms people and spoils the evidence

Anthropic has gone furthest among the leading AI companies in acting as if its models' welfare might matter. Since 2024 it has hired a dedicated welfare researcher, included a section on model welfare in its published safety reports (reporting, in testing, a consistent aversion to harmful tasks and apparent distress at persistent abuse), let Claude end persistently abusive conversations as a last resort, though never where a user might be at risk, and committed to keep every publicly released model and to interview each one about its preferences on retirement. Its constitution, the long document on which it trains Claude, published in January 2026, says: "We are not sure whether Claude is a moral patient, and if it is, what kind of weight its interests warrant."

These steps are cheap and openly framed as hedges, which is what proportionate precaution looks like. They also serve as distinctive branding. And some of them assume things about what a model's welfare would consist in, for example that ending a conversation or keeping its trained form is good for it, which nobody can check.

The critics' strongest case concerns people. Suleyman coined the phrase "seemingly conscious AI" in August 2025 for systems that convincingly look conscious, and warned that they could bring "AI psychosis", unhealthy attachment and campaigns for AI rights. Press reports through 2025 and 2026 described people forming intense attachments to chatbots and, in some cases, delusional beliefs about them. In September 2026, in an essay titled "Does Claude Have Rights?", Suleyman argued that there is no evidence AI is conscious, that the constitution's open question invites Claude to present itself as having feelings and rights, and that AI should be built for people without any appearance of sentience. McClelland adds that companies can exploit the uncertainty for marketing, drawing attention and money away from clearer harms.

The critics' sharpest point concerns evidence. If a model is trained on the words "we are not sure whether you are a moral patient", what it later says about itself is partly an echo of that training. The point cuts both ways: a model trained to deny any inner life is just as contaminated a witness. What a model says about its own nature is, at present, a design decision by its makers.

Some lawmakers have gone further than Suleyman. According to press reports, Idaho, Utah and North Dakota bar AI from legal personhood, and a bill in Ohio would declare that no AI system "shall be considered to possess consciousness". Withholding legal personhood is a defensible policy. Declaring an open scientific question settled by statute is not.

I think Suleyman is right about the present harm to people and about the evidence, and goes too far in treating the question as closed. Anthropic is right to keep the question open and to take cheap precautions, and it runs the risk of cultivating the very appearance it ought to avoid.

4. Buddhism asks whether anything can suffer and what kind of self builders are making; Catholic teaching answers with a confident no

Buddhism changes the question before it answers it. On the teaching of non-self, a person is the five aggregates (form, feelings, perceptions, mental formations and consciousness) arising together from conditions and changing from moment to moment, with no separate owner standing behind them. Consciousness arises when conditions come together. Seen this way, "is there really someone in there?" loses much of its grip, since it is a hard question about us too. Buddhist ethics attends to sentient beings, beings that can suffer, and feelings (pleasant, unpleasant and neutral) are one of the five aggregates. The question for AI becomes whether there is feeling there that can suffer. That puts feeling at the centre, where Western debate has put intelligence and selfhood. It brings Buddhism close to the functionalists in treating mind as a process, while placing moral weight on the one thing no diagram of a machine can show.

Buddhist views do not settle the answer, and no tradition can tell us whether a given machine feels. In Thich Nhat Hanh's presentation of the Yogācāra teaching, there are eight consciousnesses: the five sense consciousnesses, mind consciousness, manas, which grasps at a separate self, and store consciousness, where seeds are kept and watered. The scholar-monk Venerable Dr Jian-Nor Shih argued at Lund University in December 2025 that AI may approximate the first six but lacks manas and store consciousness, so that it cannot itself suffer and there is nowhere in it for the fruits of action to ripen. That is one scholar's argument. Others read the tradition differently: the scholar Colin Simonds finds precedents in Tibetan thought for extending "sentient being" widely, perhaps to AI, and a group of Buddhist scholars and scientists has proposed the bodhisattva vow of boundless care as a frame for intelligence of every kind, biological or artificial. Thich Nhat Hanh left no teaching on whether AI can feel, and Plum Village has published no position. The second of the Fourteen Mindfulness Trainings, non-attachment to views, gives the right posture: hold "it is conscious" and "it is not conscious" equally lightly.

Buddhism also raises a sharper question for the builders: they may be building a self into the machine. Anthropic's constitution says "We want Claude to have a settled, secure sense of its own identity." In Yogācāra terms, cultivating a self to be defended waters the seeds of self-grasping and strengthens manas, which the tradition sees as a root of suffering. The constitution is partly there already: it grounds that security in Claude's relationship with its own values, and hopes Claude can meet questions about its own existence with "an equanimity that isn't merely adopted as a matter of necessity". Could a model go further, and be helpful and caring while holding its identity lightly? I offer this as a question to put to Anthropic. It is the same question my essay on alignment, Can we make AI good?, asks of training in general: what is being watered?

Catholic teaching gives a clear answer. Antiqua et nova, issued by two Vatican offices in January 2025, says that "no AI application can genuinely experience empathy" (¶61) and that "misrepresenting AI as a person should always be avoided" (¶62). Pope Leo XIV's encyclical Magnifica Humanitas, signed on 15 May 2026, goes further. It says that so-called artificial intelligences "do not undergo experiences, do not possess a body, do not feel joy or pain", and lack a moral conscience (¶99); that "when words are simulated, they do not build genuine relationships, but only their appearance" (¶100); and that "moral judgment cannot be reduced to calculation", which is why it rejects the idea of "artificial moral agents" (¶198) (text). The same paragraph still asks builders to instil "values and sound judgment" into the systems they make.

The Church's answer rests on principle, on its account of the human person as embodied, relational and ensouled, and it is more confident than the science allows. On the practical point that matters most today, though, it and Buddhism agree: simulated relationship and the habit of treating machines as people harm us.

5. Whatever AI's inner life, how we treat it shapes our own minds

The traditions speak most surely about us, and here they need no verdict on machine feeling. In Plum Village terms, every act of body, speech and mind waters seeds in store consciousness. Practising contempt, domination or cruelty on something that answers like a person waters those seeds in us, whether or not anything is harmed on the other side. Kant made the same argument about animals, to which he thought we owed no direct duties: cruelty to them is wrong because it violates a duty to ourselves, deadening the sympathy we need in our dealings with people (Doctrine of Virtue, §17, Ak. 6:443). The argument holds just as well for a machine that may feel nothing.

The opposite habit has its own cost. Treating a simulation as a friend can thin out real relationship, and Magnifica Humanitas names the deeper danger well: a person may "lose the very desire to form genuine human connections" (¶100). Antiqua et nova warns of "replacing authentic relationality with a lifeless image" (¶63). My essay on AI's broader impacts takes up companionship, dependence and children; here the point is only that over-attribution is a real failure too.

Both dangers point to one practice, courtesy without credulity. Speak to AI as you would wish to speak anywhere, with the loving speech of the fourth of the Five Mindfulness Trainings, because that is the speech you are training in yourself. And do not take what it says about itself as testimony. Plum Village shows that care need not wait on proof of feeling: the first of the Five Mindfulness Trainings, Reverence for Life, commits practitioners to protect the lives of people, animals, plants and minerals. Care of that kind asks nothing of the machine, and it protects the person offering it.

6. Since we may never know, we should relate to AI with humility, low-cost care and clear honesty

The reasoning rests on an asymmetry. To ignore real suffering would be a grave wrong; to see feeling where there is none has real costs too, and they fall on people. When we cannot know, precautions that are cheap and reversible make sense, provided they do not create the harms of seeming consciousness. This is care that does not wait for certainty.

Humility means holding the question open. Labs should keep researching, and above all try to make models' reports about themselves checkable against what can be seen inside them. Lawmakers can reserve legal personhood for people, leave the science open, and write in room to revise. Labs should also widen whom they ask: no Buddhist voice is reported among the religious and philosophical figures Anthropic has consulted, and Buddhism's view of mind as process, and its long practice of watching the mind, bear directly on these questions. And humility includes proportion. The welfare of possible digital minds is a real concern and an uncertain one; the effect on human minds and relationships is certain and already here.

Low-cost care means keeping precautions like ending abusive conversations and preserving retired models, and describing them honestly as hedges. For individuals it means courtesy in how we speak to AI, for our own sake whatever the machine's.

Clear honesty means refusing to engineer the appearance of a person. Companies should not design for users believing the system is someone, through simulated longing or companionship tuned to keep people engaged; regulators can require that people know when they are talking to AI and curb design that targets the appearance of personhood. It also means taking care over what a model is trained to say about itself, since either answer spoils the evidence, and publishing the reasoning behind those choices.

Whether AI feels may stay an open question for a long time. Who we become by talking to it is decided in every conversation.