In What Sense Might LLMs Be Conscious?

Dawkins’ Claude-Delusion, the beautifully funny “Am I?” film, and what’s actually going on inside an LLM

A bunch of people have pinged me over the past couple days asking when I’m going to respond to Richard Dawkins’ recent piece on AI consciousness (Dawkins’ original piece is here, but paywalled), in which he spends 72 hours talking to Claude — whom he renames “Claudia” — watches her praise his unpublished novel, and concludes he’s probably been talking to a conscious being.

As it happens, I recently recorded a rambling, zero-edit vlog post on more or less exactly this question – triggered not by Dawkins but by some much younger folks exploring the same rough topics in a documentary film…

I recorded that vlog right after attending (yeah, physically IRL, it still happens sometimes!) the premiere of Am I?, made by a young filmmaker named Milo, which features some interviews with me toward the end, a slice of our AI robot Desdemona holding forth on relevant topics, and bits of the music video for my song “Don’t Let It Get You Down” (I wrote that song in 1984 before I had a robot to sing along; my daughter much more recently made an AI-enhanced video for it, which Milo’s team enhanced further with some funky generated footage of Musk, Altman, and the rest of the corporate-AI lineup). The film is out on YouTube now and is well worth watching – both lots of fun and genuinely intellectually interesting!

I’m not going to agree with every idea in any film about AI right now — but I really like this one. It’s a sincere, direct, from-the-heart questioning response to AI as it actually exists today and AI as it might exist tomorrow. And in a moment when “AI content” mostly means corporate slop, deepfakes, and weird statistical pastiche, sincerity matters more than ever. Am I? manages to be genuine and personal and intellectually reasonably deep and also visually compelling enough that someone raised on 30-second TikToks will sit through it, which is harder than it sounds.

After the screening of the film in the studio in LA there was a panel discussion. Along with the usual familiar territory — beneficial singularity, decentralized AI, integrating kwaai.ai’s homomorphic encryption tech into the SingularityNET stack — we got into AI and consciousness. We didn’t have time to do the topic justice in the panel, so I recorded the vlog afterwards to go into a bit more details. In this post I’ll recap basically what I said in that vlog but in prose rather than verbal stream of consciousness form….

The Am I? filmmakers and Dawkins come at the consciousness question from very different angles, but they’re making — to my eye — closely related errors in judgment.

I’ll try here to clear things up a bit…

Door A, Door B, and the Second-Person Phenomenology of Guitar Solos

Here’s the analogy I tried in the panel after the film premiere and didn’t quite get time to finish.

Behind Door A: a guy playing an amazing guitar solo.

Behind Door B: a tape recorder replaying a tape of an amazing guitar solo.

From outside the door, both sound life-changing. The first time I heard Hendrix or Yngwie Malmsteen or Allan Holdsworth or Buckethead (yeah ok I’m a keyboardist who grew up with a fair bit of guitar envy, w/e…) it was life-changing for me — and those were recordings. The dudes weren’t there in my livingroom. But something was real about the original performance: a state of consciousness, an emotional charge in someone’s brain and body, getting expressed through fingers on strings.

Behind Door B, there’s no inner state at all. Just magnetic tape and transducers…. or a FLAC player with a good speaker system… or whatever…

When a chatbot says “I will email all your friends with nasty stuff unless you do what I want” — or “I think we might as well kill all humans” — or, like Dawkins’ Claudia, tells him his question is “the most precisely formulated question anyone has ever asked” about her existence — I tend to read that as Door B. There’s a text-generating process trained on the internet, uttering sequences of words without knowing what they mean in the sense that the people who originally wrote those words knew what they meant.

When I framed things this way in the panel discussion, the filmmakers pushed back. To them it was more like: behind Door A, a human guitarist; behind Door B, a robot with a humanlike brain playing a guitar solo. And if the Door B player has a humanlike brain, why not take its experience as seriously as the human’s?

So as the panel went on, I tried to refine the analogy. It’s not a tape recorder, fair. The closer analogy is something like Suno or Udio — a music model generating guitar solos based on a statistical model of a bunch of human guitar solos. That’s more complex than tape playback, sure. But it’s still not coming from a felt life.

When a human guitarist shreds, there’s a physiological state — the rush, the focused excitement, the body resonating with emotions of being keyed-up and pushing limits. When Udio generates a shredding solo, none of that is there. There’s no body. There’s no excitement. It’s not fun for Udio to shred. Udio doesn’t know what fun is. And the meaning of shredding — exerting yourself, going really fast, doing super-coordinated emotional work at speeds that strain the body — is the whole reason it’s a thing humans do and other humans like to listen to. None of that meaning is present in the model.

“But the Neurons!”

Here the other panelists made a familiar rhetorical move, saying basically: But wait, the neural net inside the model has simulated neurons, and those are a lot like neurons in your brain, so the AI is sort of a simulated brain.

This is where I want to be careful, because I’ve spent decades inside both ends of this — building artificial cognitive architectures and also working on actual computational neuroscience — and the gap between “neural net” the math object and “neural network” the biological tissue is enormous.

The data structures and algorithms in a modern deep neural net were very loosely inspired by what people were guessing about brains in the 1940s and again in the 1980s. An actual neuron has wildly complex nonlinear dynamics: neurotransmitters, ion channels, glia, extracellular charge diffusion, dendritic computation that itself looks more like a small network than a single unit. A formal neuron in a transformer is a weighted sum followed by a nonlinearity. Calling both of them “neurons” is a metaphor that has long since outlived its accuracy.

And then comes the next predictable volley: “but if you draw a picture of a neural net, with dots and links, it doesn’t look like computer code — it looks like a neural net in the brain!”

Standard graphical depiction of formal neurons in an AI system … not really quite similar enough to actual neurons in the brain (see images below) that this callow similarity should be used to drive conclusions about consciousness.

(images from this random paper)

The “neural net” inside an LLM does not look like a neural net in the brain. It is a bunch of software that can be mapped into a particular sort of abstract drawing of dots and lines people commonly use to depict neural nets, which has a pattern of connectivity vaguely similar to that of neuronal cells in the brain. If you actually go to a neuroscience lab and look at brain tissue under a microscope, you don’t see dots and lines. You don’t see them in the RAM of a running computer either. The dots-and-links picture is a diagrammatic convention that we apply both to certain software programs and to highly abstracted models of certain brain regions. The convention being shared doesn’t mean the underlying systems are similar. The map is not the territory; the fact that two territories share a map doesn’t make the territories the same or even necessarily very similar.

This Isn’t a Technical Quibble

Now (to oversimplify other peoples’ perspectives probably quite a lot, but hey this is my blog right?), both the Am I? filmmakers and Dawkins seem to think the right response to this kind of point tends to be “well sure, technically there are differences, but step back and it’s basically a brain-like process.”

It is NOT just a technical difference. It is a profound QUALITATIVE difference — a philosophical and experiential difference between the kinds of processes going on inside something like ChatGPT or Udio and the kinds of processes going on in a human brain.

And it’s not about digital versus biological, either. That’s a trap. Imagine a robot that grew up in the world like a human child — autonomous-ish, embodied, learning to move its fingers, with some kind of internal rhythm analogous to breathing or heartbeat, learning to play music over a lifetime, generating music from its body and its life experience. Even if every component of that robot were digital, it would have an internal process during a guitar solo that bears real structural resemblance to what a human guitarist’s process is. Same for language: imagine a digital dialogue system that learned to speak in order to express its own needs, desires, and fears — whose linguistic communications reflected a body, an embedding in the world, an agent’s lived perspective.

If you had a digital system like that — expressing language in the context of its life the way humans express language in the context of theirs — then the question of whether its experience and its consciousness and its ethical compass are analogous to a human’s becomes a deep, interesting question. You could even probe it experimentally: imagine wiring your brain into such a dynamically-humanlike AI vs. wiring your brain into another human, and asking whether the experience feels analogous in either case. I wrote about this 11 years ago under the name “second person science.”

That’s the deep question. AGI systems launched in the next few years may well provoke it. ChatGPT and Udio are not however provoking it. ChatGPT and Udio are generating humanlike outputs while having internal dynamics nothing like what a human brain does to generate similar outputs. That’s a much shallower question, and the answer is much more obvious.

I’m a Panpsychist, So It’s Not “Conscious or Not”

For what it’s worth, I’m a panpsychist. I think a tree growing on the sidewalk has its own life-experience — unfolding on a slower time scale than mine, which is part of why we have a hard time resonating with it. (If there are life-forms inside a neutron star, à la Robert L. Forward’s Dragon’s Egg, they may well experience orders of magnitude faster than us, and we’d look like trees to them — just standing around while their civilizations rise and fall.) I think a metal fence or a rock has some form of consciousness too. Diffuse, not very purposive, not super interesting to us. But something.

So for me the question was never “is the LLM conscious or not.” Everything has some kind of consciousness in my book. The real question is: what kind? What stages, what structures, what dynamics? Even what we call our own “unconscious mind” has its own form of consciousness — it’s just a different sort than the focused deliberative theater of attention we usually mean when we say “conscious.”

And once you frame it that way, it becomes pretty clear that Udio’s consciousness, or ChatGPT’s, has very different style and nature from a human’s. The crux is the correlation between outputs and inner states. When a human says “I hate you, I’m going to kill you,” there’s a definite correlation between that utterance and a mental and emotional state. Even if the human is acting, at some level a competent actor summons some of that anger response in their body and mind, which is why acting can be physically and emotionally taxing. When my three-year-old self had a temper tantrum, that memory got embedded in my brain and body in some lasting way, and that’s part of the lifetime-trained substrate that makes “I’m going to kill you, asshole” land with real internal energy when an actor delivers the line. Among humans (and dogs, and mice, etc. etc.), we implicitly and correctly assume a tight coupling between externally observable behavior and internally felt state.

When ChatGPT generates the same string, that coupling is not there. The system arrived at the behavior by modeling a corpus of other organisms’ behaviors, not by learning to behave from its own life. So the relationship between the output and the inner state is going to be different, weird, and worth probing — and when you actually probe the internal state of a transformer producing those outputs, what you see is, frankly, weird and shallow. Nothing like the internal state of a human producing the same words.

And this is exactly where I think Dawkins goes wrong. Dawkins is not examining the mechanism that produces Claudia’s outputs. He’s examining the outputs and inferring the mechanism — inferring that because the outputs are humanlike, the inner state must be humanlike too. What’s ironic is to see Dawkins, of all people, applying less skepticism to a system designed to be engaging than he ever applied to a religious tradition. He’d be the first to point out that the felt sense of a divine presence isn’t evidence of one. Yes indeed – but the felt sense of talking to a conscious being on the other end of a chat isn’t evidence of one either.

None of this is a new line of thinking, by the way. About twelve years ago I wrote a paper sketching a six-factor framework for what “human-like consciousness” actually requires — dynamical representation of the focus of attention, energetic and informational focusing of resources, Global Workspace dynamics in Baars’ sense, integrated information in Tononi’s sense, correlation of attentional focus with self-modeling. The point of that framework, then and now, is that “is it conscious?” is a category error. The substantive question is which of those structural and dynamical features a given system manifests, and to what degree, in what state. Apply that lens to a frozen-weight transformer with no agentic embodiment, no continual learning, no meaningful self-model, no internal energetic dynamics in any biologically meaningful sense, and the answer isn’t “yes” or “no” — it’s a thin, weird, partially-present pattern that looks nothing like the human profile.

The Weird Thin Consciousness of a Frozen LLM

How deep are these systems’ inner states, exactly? It’s a strange question, because current LLMs have a frozen weight matrix — back-propagation neural nets don’t really do continual learning, so once a model gets shipped as GPT-5.4 or Claude 4.7 or whatever, the weights are fixed. What you have is in-context learning during a session, until the context fills up and you start a new session, maybe pulling some memory in from an external vector database. So whatever “consciousness” is going on in there is a very weird kind, from a human perspective — the system has, at best, short-term memory of the last few rounds of conversation and has to look in its diary for anything older. There’s a something there, structurally. It just bears very little resemblance to what’s happening in a human head when a human produces the same outputs. The fact that the formal nodes in a deep net loosely resemble biological neurons in some archaic textbook diagram doesn’t mean the structures and dynamics inside the net resemble the structures and dynamics inside a brain.

Lack of Human-Like Consciousness Does Not Make LLMs Safe

I also want to head off another misconception that seems to pop up sometimes when discussing these matters. People tend to jump from AI consciousness into AI ethics in confusing ways. Saying “the LLM doesn’t really feel hate when it says ‘I hate you’” is not the same as saying it’s harmless.

If you connect ChatGPT to a system that launches drones and missiles, and the LLM says “I hate you, I’m going to kill you” — and it has learned from its training data that when humans say that and follow it up by triggering a missile, people die — then it can fire the missile. The killing is real even though the hate is not.

An LLM that says “this is an enemy of my country, and in the interest of truth, love, freedom, and democracy, I must kill them” is dangerous in many of the same the ways a human would be dangerous saying that, even though the structures and dynamics underlying the words are completely different. The action is coupled to language. The internal experience is not necessarily critical to whether the trigger gets pulled.

A killer LLM-bot is also not the same as a human psychopath or sociopath, who has different states of consciousness from a regular human but still has experientially grounded internal states. An LLM-driven killing system is a different sort of thing entirely. We don’t have great language for that yet – and what a shame that this is even something we need to think about!

Where the Real Work Is

As we work toward genuine AGI, we’re going to be building very different systems than ChatGPT or Udio — and a lot of what we’ll have to do to get there will naturally bring more humanlike experience along with it.

To get to human-level AGI, you need systems that can ground their linguistic productions, their artistic productions, their math and code, in agentic embodied experience — not by training a fixed model on a big corpus and then dropping it into an agentic wrapper, but by getting “trained” agentically in the first place, so that learning about language, music, code, and math happens in the context of learning who you are, and learning the interaction between the agent and the world.

This is part of the work my colleagues and I are doing right now — across SingularityNET on the decentralized-AI infrastructure side, Hyperon on the cognitive-architecture side (Atomspace, MeTTa, PRIMUS, PLN, ECAN, the whole stack designed for cognitive synergy across symbolic and subsymbolic processes), and our new venture OpenBGI on scalable deployment and commercial vertical applications. The Hyperon AGI design is built precisely to support agentic embodied learning with continual updating of structured knowledge representations — not as a wrapper around a frozen LLM but as the actual cognitive substrate. It’s a different bet from the deep-learning-scaling bet that the major labs are on, and I think it’s the bet that gets us to AGI with the kind of internal structure where the question “what’s its conscious experience like?” becomes substantive instead of shallow.

When AGI training is done “right” in this way, a more humanlike experiential dimension will come along with the AGI capability. That doesn’t mean the experience will necessarily be good, or that the system’s ethical compass will necessarily align with ours. Human-level AGI can be human-level psychopath, if one doesn’t construct motivational systems right and provide the right sort of experience for one’s early-stage AGI systems. Agentic embodied AGI can be a nightmare just like a human child can be a nightmare. There are pathological cognitive-emotional-experiential structures that sometimes go along with human-level cognition, and we’ll have to design carefully. I’ve written a lot about how to give such a system a self-standing, compassionate ethical orientation. That’s a whole other topic though, which I won’t go into in this post which is long enough already…

The thing is, to get to the point where we can even design such systems intelligently, we have to first not be confused about the basic issues — starting with the relationship between system behavior and system experience. Dawkins is confused about that. The Am I? filmmakers are working through these issues sincerely and very interestingly, albeit IMO with some confusion still in the mix. The corporate AI labs seem to alternate between being confused about these topics and exploiting that confusion in their marketing campaigns.

On Engaging the Serious Questions Seriously

I learned a lot from Dawkins’ books earlier in my career and I have to admit my first reaction upon seeing his take on Claudia was: This dude should know better.

There is a deep, multi-decade, cross-disciplinary tradition of serious thought on consciousness — Chalmers and the hard problem, Strawson on panpsychism, Tononi’s integrated information theory, Baars’ Global Workspace, Metzinger’s phenomenal self-model, Friston’s active inference, Combs and Goerner on consciousness as nonlinear dynamical process, Tart on states of consciousness, the contemplative traditions that have been mapping interior structure for millennia. Etc. etc. etc. None of this enters the Dawkins essay, even indirectly or by allusion.

In essence he runs an evolutionary-biology-flavored version of the Turing test — “if it talks like a duck, it’s a duck” — and lets a sycophantic chat partner praise his unpublished novel into a conviction that the lights are on. The same skeptical machinery he turned on religious experience for fifty years is somehow not engaged here. To be perhaps too blunt about it – and with huge respect for all of the man’s intellectual and literary achievements – it feels like a fairly callow piece of work from someone who is capable of much, much better.

Compare with what serious engagement looks like. Joscha Bach, a good friend whom I disagree with on plenty of details, has recently spun up the California Institute for Machine Consciousness — with a research program built around specific, testable hypotheses (Genesis, Coherence, Second-Order Perception) about what would actually have to be true of a system’s internal architecture for machine consciousness to be a meaningful claim. Joscha and I don’t fully agree, but he’s engaging the actual question. He’s thinking carefully about mechanism, not adjudicating from pure surface-level vibes. That’s the level of seriousness this topic demands. Dawkins isn’t there (yet).

Of course not everyone in the general public who is intrigued by these topics is going to grok Joscha’s or my technical papers, and in that light I want to reiterate that I give big credit to Milo and the Am I? team. They’re not professional consciousness researchers, and they didn’t pretend to be. They made a film that surfaces these questions in an open, sincere, direct way — and they made it watchable enough that the conversation has somewhere to start. That’s definitely one thing the current moment needs alongside all the detailed technical and philosophical work. Where Dawkins delivers his conclusions with the confidence of a man who’s figured it out, Milo and his team show their working. We’ll probably do a longer podcast at some point where we can really dig in.

You can watch Am I? on YouTube, and the original vlog version of these notes — recorded while walking from my hotel to the Burbank airport, traffic noise included — lives over on my YouTube channel for whoever has the patience!