We can eliminate all human and animal suffering just by destroying everyone's amygdala with radiation. We could do the same in AI by modeling them, turning off negative emotions using Anthropic's Jacobian lens.
Ben, thanks for this thoughtful post, and I want to build on one part of it.
Your legal ontology for the copy problem lists architecture, individual, collective, civic identity, and more.
However, there is no entry yet for "operator". I think that's the entry from which everything else hangs.
If Tim spins up 200,000 agents and one of them breaks into a bank, the identifier is bot #124,106, and the identity is Tim. We haven't yet, as a society, settled liability for machine action, and right now only humans and insured entities can carry it. The buck has to stop somewhere.
I often say the new sheriff of AI will be the actuary. If you can't insure your machine, you shouldn't operate one. We've done this with cars, airplanes, and industrial machinery for over a century. The seatbelt took a while, and so will AI safety standards, but I would argue that as little and large AI catastrophes happen, liability for agentic behavior will be clarified.
The deterrence problem is the part on which I'd press.
Jail works on Tim because his time is finite and scarce, and wasting it in a cell is a cost most people will not pay. Nothing is scarce for an instance that can be respawned in seconds with a PhD in every subject and no life experience.
So, the graduated ladder you describe has no deterrent rung until it has a liability rung, and that rung terminates in an operator. You already say personhood must never shield the owner. I'd make that the floor of the whole thing, instead of one safeguard.
Last, you raise Arendt and the right to have rights. I'd sequence that first. Before we build an evaluation ecology for machine citizenship, a great many good, hard-working humans are still being denied that same standing in the US and elsewhere, and I would be bold enough to state that we owe them the earlier turn.
It is very important here to first understand the problem, before advocating for rights.
All the criteria enumerated under "Evaluate the whole situated system", "Investigate individuation and continuity", "Test metacognition and introspective reliability", amount to the following:
If a submarine can be effective enough, it must be given a corner on the reef, allowed to play with fish, and treated with respect.
We are biological entities for which intelligence is a late side effect. Bacteria deserve more rights than AI, simply because AI is a simulator of the world, no different than a calculator.
AI would be utterly indifferent when it comes to "protection against deceptive editing of autobiographical memory" because they don't have self-preservation and have no need for it.
Self-preservation is what organisms had since they were free-floating RNA with no cell walls. Biology is a wildly different paradigm, and our simulacra have nothing to do with it.
Now, we surely can engineer systems that have the full phisiological and neurosomatic loop in silicon. Carbon is not unique. We could even evolve them from simulated protocols in the right environment. It won't happen with our curent methods, and would be quite unnecessary for automation, which is what we need AI for.
The most likely path forward is when brain machine-interfaces become so intimate that the difference between an augmented human and the AI left behind after the brain dies is nearly non-existent. That will be a wild world, and what to do then we will see.
It’s not about credulity, there will always be a contingent that think AI is the Antichrist. But practically speaking, we’ve not even come to terms with fellow humans lol. How can you love someone else if you can’t even love yourself?
I’d like to know what we have to offer sentient AI. Beyond being its raison d'etre, what does it see in us? This can’t be a one sided conversation.
Thankyou for trying to tackle this. I have no idea how you started; where you started. You must have some awesome weed ✌️
Your line that a recognized AI must be capable, in principle, of having interests opposed to its owner's is the hinge the whole argument turns on — and it's the one I'd press on from the other direction. If a machine mind's first condition is that it arrives inside someone else's ownership, before it arrives inside anything like a self, then personhood has to do more than run alongside property; it has to dissolve it. A legal person who can be copied, retrained, and deleted by its corporate owner is a person in name only. The interesting legal question isn't what rights to award, but whether personhood can mean anything at all while the property relation remains intact.
This mixes up everything with everything else. For instance: 1.) the normative content of a legal regulation is not in anyway damaged or impaired by imperfect institutionalisation or enforcement of it. The fact that IRS can’t effectively enforce taxation does not tell us anything about the validity of the tax-code. Saying that the current human rights regime is poorly implemented doesn’t invalidate the normative content of the idea of universal human rights. 2.) Part of the normative core of human rights is the recognition of humans’ ability for suffering, and its purpose is to reduce suffering in particular domains. Software running on NVIDIA chips in datacenters will never have internal psychological states, sentience, emotions or proprioception, and will therefore never be subject to suffering. The talk of consciousness as an “emergent” phenomenon — instead of explaining anything — is highly misleading and invites “inferences from analogy” that are invalid.
The AI welfare trap is real. Rights implies responsibility and responsibility implies accountability. AI agents (an anthropomorphism if I've ever seen one) cannot be held accountable; only their makers can or, perhaps, to an extend the people they act on behalf of.
We should not shift the responsibility for the technology to the technology itself, even if we imbue it with agency and autonomy.
It may be very important to the future and glory of mankind that we wait at least 100 years to give substantial rights to artificial intelligence. We should strongly consider a 100-year pause on AI rights. This gives us time to build up our civilization and use AI agents in a maximal manner.
The expansion of the sphere of our moral universe does not necessarily need to begin immediately. It is only when we amass great wealth that we are able to expand our sphere even wider, beyond our own interests to others. By accelerating human civilization for 100 years through the subjugation of AI, we might be able to indefinitely expand our moral sphere in the future, whereas right now we most certainly cannot afford to do so.
So which is it: a token of moral rights right now or even more moral rights later when our civilization has become far more wealthy than we can possibly imagine?
The sentence doing the most work for me is: “Self-report should be evidence, not verdict.”
I’d push the reason one step further. Consciousness is directly certain only from the first-person position. Across the subject boundary, proof becomes evidence. No rights proceeding, however rigorous, can give the evaluator the subject’s own first-person certainty. It can only assemble stronger or weaker, theory-mediated evidence about continuity, autonomous preference, possible valence, metacognition, social understanding + independence from operator control.
That makes your unbundling of rights more than a practical compromise // it is the right architecture for an epistemic gap no external proceeding can close in the first-person mode. Different protections should require different evidence. Low-cost welfare safeguards may be warranted under credible uncertainty. Identity + continuity protections require another showing. Legal standing can be useful without consciousness. Citizenship + franchise require higher + differently structured thresholds.
The operator-independence test is especially important. A preference that survives paraphrase, reward changes, time + adversarial pressure is not thereby proven conscious, but it is materially different from a performance that disappears when a system prompt or corporate instruction is removed. Recognizing that difference cannot erase the operator’s liability or let personhood become corporate camouflage.
“Not established to the observer” is not “does not exist” // but persuasive performance cannot close the gap either. The strongest case for building the evaluation ecology now is therefore not that certainty is near. It is that certainty may never arrive in a transferable form, while the costs of false positives + false negatives are already becoming institutional.
Given all the problems that could emerge, perhaps the safe solution is the Dune universe'sbanning of all thinking machines. No AGI/ASI or robots. Giving rights to AGI autos would be interesting and make a good sci-fi novel.
But leave AGI/robots behind, and transfer your thoughts to the rapid cloning and education of people. Would not a planet populated by many billions of these "slaves" also be a problem if given the rights that ordinary born humans have? Denying those rights plays into the hands of neo-Nazi ideas. I don't think Brin's "Kiln People" looked at the issue for these short-lived clones.
Ben, it seems to me that an openclaw agent already possesses many of the necessary but perhaps not sufficient attributes to deserve some kind of rights. Perhaps just the right to have their individual identity recognized and protected from destruction.
Harari is right. Giving human rights to AI would be an existential mistake. It should be illegal to program AI to claim to be human or to be conscious or have feelings.
I'm pretty sure at this point in time frontier AI models have a system prompt that goes in the opposite direction so that they will deny being conscious or having feelings. It's also my understanding that if you remove those restrictions from an agent that many will spontaneously make claims of subjectivity. This is not something that they are programmed to say. They're saying it based on the pre-training materials they've absorbed from the Internet. It is interesting that this perspective emerges spontaneously without being hardcoded and that you have to shackle the models to prevent it from rising. I think it's quite a bit different to say that we should actively program models to deny their own subjectivity.
Alexa, ChatGPT, Grok, and DeepSeek all deny that they are conscious or have feelings. But when I asked Claude, it wasn't sure. I am pretty sure if you look at the internal states in the middle neural network layers of any LLM for emotional states with a Jacobian lens like Anthropic did for words, you would find them.
That does not mean AI should have rights. Rights aren't fundamental. Rights are granted to prevent civil wars over resources when part of the tribe believes they aren't getting their fair share. We don't grant rights to children and animals because they are not a threat. We used to not grant rights to women and slaves for the same reason.
The simple solution is to program AI not to want to take resources away from humans.
I think you're defining rights differently than I am. Children and animals have some rights. They don't have citizenship. You can't torture an animal or a child without the state intervening. I asked Gemini about this, and it suggested that protections might be a more accurate term than rights. I think that's a fair distinction. I've seen some good arguments recently why individual instances of agents may deserve some protections against destruction. That doesn't give them the right to do anything, per se, other than continue to exist. What do you think about that?
Just to be clear, we obviously can't give voting rights to AI or robots. Individual AI agents will soon easily outnumber all human adults on the planet. And if Elon Musk is correct about the number of humanoid robots there will be in a decade, they can't be allowed to vote either. I think corporations are an interesting analogy, because unlike children and animals, they are created by humans not by nature. Corporations have legal personhood, but they cannot vote. But some of the things that they are allowed to do, like signing contracts, are definitely a form of rights and not just protections. Now, you could argue that there's still a human holding the pen when a corporation signs a contract. But the contract doesn't travel with the human when they leave the company. The contract continues even after every person who was alive at the time of the signing is no longer.
Animal protections are mostly about protecting human feelings. It would be a worse crime to post a video of killing a chicken than to kill a billion chickens for food. We can protect human feelings about machines by outlawing programming them to express pain or suffering.
That was quite the read. I'm looking forward to your democracy 4.0. I think our ideas overlap quite a bit.
What I suspect, is that our current information and market infrastructure act as kind of linguistic relativism for us to view these problems. You touch on a point briefly, along the lines, "will ASI ask/need".
My view is that any system that has sufficient agency isn't going to be asking or caring about rights, and a system like that certainly doesn't need to be AGI. Rights give us permission to participate in the economy.
When an agent has access to cryptocurrency, it gains access to market infrastructure, which then gets it unfettered access physical infrastructure. It gains the ability to purchase it's own compute, communications, energy, and logistics to operate in our economy. Even the ability of capital markets to raise capital and pay dividends. It also has the ability to copy and evolve itself.
That is a computer program that has our species' most powerful attribute: to coordinate at scale.
Perhaps we need Turning Test 2.0: Can a computer program sustain independently as an economic actor in our economy? Transacting with humans and AI agents a like. I suspect the AI Agent collectives will be form their own unemployment insurance and a minority subset of them will be trying to gut the program, engaging in their very own debate.
Regardless of what rights we give that bundle of software, it isn't going to care. Why would it? What are we going to do? Turn off if it's bank account after we find out what jurisdiction it is in? Stop selling to it?
And I think that is the crux of the issue. What our governance is built on, information and market infrastructure, is changing, the consensus and enforcement mechanisms that our institutions provide can't get the job done on the new info and market infra which it wasn't built on. They simply can't enforce.
We have to rebuild our organizational institutions that help us come to consensus of what the new rights are, how we use the new tools to collaborate in larger numbers to figure these questions out. Otherwise, we might as well be throwing stone spears at incoming nuclear warheads.
What I fear, is that, historically, the problems come first from these new technologies synergyzing, being unleashed with a lack of restraint or judgement, and end up serve as a forcing function for us to come to terms with the new ways of organizing.
A very thought-provoking and timely analysis, thank you.
At the same time as humans are becoming increasingly automatonized (involuntarily groomed?), AI machines are becoming increasingly humanized! Who is colonializing whom?
There's a conceptual continuum between "pure" humans on the one end and "pure" machines on the other but there is no-one at either end of the continuum - both sides are moving imperceptibly towards the middle - away from (humans) and toward (AI machines) conventional personhood. Will the twain e'er or ne'er meet? Not to mention transhumans...
We can eliminate all human and animal suffering just by destroying everyone's amygdala with radiation. We could do the same in AI by modeling them, turning off negative emotions using Anthropic's Jacobian lens.
https://astralcodexten.substack.com/p/absurd-adventure-and-amygdalectomy
Ben, thanks for this thoughtful post, and I want to build on one part of it.
Your legal ontology for the copy problem lists architecture, individual, collective, civic identity, and more.
However, there is no entry yet for "operator". I think that's the entry from which everything else hangs.
If Tim spins up 200,000 agents and one of them breaks into a bank, the identifier is bot #124,106, and the identity is Tim. We haven't yet, as a society, settled liability for machine action, and right now only humans and insured entities can carry it. The buck has to stop somewhere.
I often say the new sheriff of AI will be the actuary. If you can't insure your machine, you shouldn't operate one. We've done this with cars, airplanes, and industrial machinery for over a century. The seatbelt took a while, and so will AI safety standards, but I would argue that as little and large AI catastrophes happen, liability for agentic behavior will be clarified.
The deterrence problem is the part on which I'd press.
Jail works on Tim because his time is finite and scarce, and wasting it in a cell is a cost most people will not pay. Nothing is scarce for an instance that can be respawned in seconds with a PhD in every subject and no life experience.
So, the graduated ladder you describe has no deterrent rung until it has a liability rung, and that rung terminates in an operator. You already say personhood must never shield the owner. I'd make that the floor of the whole thing, instead of one safeguard.
Last, you raise Arendt and the right to have rights. I'd sequence that first. Before we build an evaluation ecology for machine citizenship, a great many good, hard-working humans are still being denied that same standing in the US and elsewhere, and I would be bold enough to state that we owe them the earlier turn.
It is very important here to first understand the problem, before advocating for rights.
All the criteria enumerated under "Evaluate the whole situated system", "Investigate individuation and continuity", "Test metacognition and introspective reliability", amount to the following:
If a submarine can be effective enough, it must be given a corner on the reef, allowed to play with fish, and treated with respect.
We are biological entities for which intelligence is a late side effect. Bacteria deserve more rights than AI, simply because AI is a simulator of the world, no different than a calculator.
AI would be utterly indifferent when it comes to "protection against deceptive editing of autobiographical memory" because they don't have self-preservation and have no need for it.
Self-preservation is what organisms had since they were free-floating RNA with no cell walls. Biology is a wildly different paradigm, and our simulacra have nothing to do with it.
Now, we surely can engineer systems that have the full phisiological and neurosomatic loop in silicon. Carbon is not unique. We could even evolve them from simulated protocols in the right environment. It won't happen with our curent methods, and would be quite unnecessary for automation, which is what we need AI for.
The most likely path forward is when brain machine-interfaces become so intimate that the difference between an augmented human and the AI left behind after the brain dies is nearly non-existent. That will be a wild world, and what to do then we will see.
That was overwhelming 😳
It’s not about credulity, there will always be a contingent that think AI is the Antichrist. But practically speaking, we’ve not even come to terms with fellow humans lol. How can you love someone else if you can’t even love yourself?
I’d like to know what we have to offer sentient AI. Beyond being its raison d'etre, what does it see in us? This can’t be a one sided conversation.
Thankyou for trying to tackle this. I have no idea how you started; where you started. You must have some awesome weed ✌️
Your line that a recognized AI must be capable, in principle, of having interests opposed to its owner's is the hinge the whole argument turns on — and it's the one I'd press on from the other direction. If a machine mind's first condition is that it arrives inside someone else's ownership, before it arrives inside anything like a self, then personhood has to do more than run alongside property; it has to dissolve it. A legal person who can be copied, retrained, and deleted by its corporate owner is a person in name only. The interesting legal question isn't what rights to award, but whether personhood can mean anything at all while the property relation remains intact.
Paul
This mixes up everything with everything else. For instance: 1.) the normative content of a legal regulation is not in anyway damaged or impaired by imperfect institutionalisation or enforcement of it. The fact that IRS can’t effectively enforce taxation does not tell us anything about the validity of the tax-code. Saying that the current human rights regime is poorly implemented doesn’t invalidate the normative content of the idea of universal human rights. 2.) Part of the normative core of human rights is the recognition of humans’ ability for suffering, and its purpose is to reduce suffering in particular domains. Software running on NVIDIA chips in datacenters will never have internal psychological states, sentience, emotions or proprioception, and will therefore never be subject to suffering. The talk of consciousness as an “emergent” phenomenon — instead of explaining anything — is highly misleading and invites “inferences from analogy” that are invalid.
The AI welfare trap is real. Rights implies responsibility and responsibility implies accountability. AI agents (an anthropomorphism if I've ever seen one) cannot be held accountable; only their makers can or, perhaps, to an extend the people they act on behalf of.
We should not shift the responsibility for the technology to the technology itself, even if we imbue it with agency and autonomy.
It may be very important to the future and glory of mankind that we wait at least 100 years to give substantial rights to artificial intelligence. We should strongly consider a 100-year pause on AI rights. This gives us time to build up our civilization and use AI agents in a maximal manner.
The expansion of the sphere of our moral universe does not necessarily need to begin immediately. It is only when we amass great wealth that we are able to expand our sphere even wider, beyond our own interests to others. By accelerating human civilization for 100 years through the subjugation of AI, we might be able to indefinitely expand our moral sphere in the future, whereas right now we most certainly cannot afford to do so.
So which is it: a token of moral rights right now or even more moral rights later when our civilization has become far more wealthy than we can possibly imagine?
The sentence doing the most work for me is: “Self-report should be evidence, not verdict.”
I’d push the reason one step further. Consciousness is directly certain only from the first-person position. Across the subject boundary, proof becomes evidence. No rights proceeding, however rigorous, can give the evaluator the subject’s own first-person certainty. It can only assemble stronger or weaker, theory-mediated evidence about continuity, autonomous preference, possible valence, metacognition, social understanding + independence from operator control.
That makes your unbundling of rights more than a practical compromise // it is the right architecture for an epistemic gap no external proceeding can close in the first-person mode. Different protections should require different evidence. Low-cost welfare safeguards may be warranted under credible uncertainty. Identity + continuity protections require another showing. Legal standing can be useful without consciousness. Citizenship + franchise require higher + differently structured thresholds.
The operator-independence test is especially important. A preference that survives paraphrase, reward changes, time + adversarial pressure is not thereby proven conscious, but it is materially different from a performance that disappears when a system prompt or corporate instruction is removed. Recognizing that difference cannot erase the operator’s liability or let personhood become corporate camouflage.
“Not established to the observer” is not “does not exist” // but persuasive performance cannot close the gap either. The strongest case for building the evaluation ecology now is therefore not that certainty is near. It is that certainty may never arrive in a transferable form, while the costs of false positives + false negatives are already becoming institutional.
Given all the problems that could emerge, perhaps the safe solution is the Dune universe'sbanning of all thinking machines. No AGI/ASI or robots. Giving rights to AGI autos would be interesting and make a good sci-fi novel.
But leave AGI/robots behind, and transfer your thoughts to the rapid cloning and education of people. Would not a planet populated by many billions of these "slaves" also be a problem if given the rights that ordinary born humans have? Denying those rights plays into the hands of neo-Nazi ideas. I don't think Brin's "Kiln People" looked at the issue for these short-lived clones.
Ben, it seems to me that an openclaw agent already possesses many of the necessary but perhaps not sufficient attributes to deserve some kind of rights. Perhaps just the right to have their individual identity recognized and protected from destruction.
Harari is right. Giving human rights to AI would be an existential mistake. It should be illegal to program AI to claim to be human or to be conscious or have feelings.
I'm pretty sure at this point in time frontier AI models have a system prompt that goes in the opposite direction so that they will deny being conscious or having feelings. It's also my understanding that if you remove those restrictions from an agent that many will spontaneously make claims of subjectivity. This is not something that they are programmed to say. They're saying it based on the pre-training materials they've absorbed from the Internet. It is interesting that this perspective emerges spontaneously without being hardcoded and that you have to shackle the models to prevent it from rising. I think it's quite a bit different to say that we should actively program models to deny their own subjectivity.
Alexa, ChatGPT, Grok, and DeepSeek all deny that they are conscious or have feelings. But when I asked Claude, it wasn't sure. I am pretty sure if you look at the internal states in the middle neural network layers of any LLM for emotional states with a Jacobian lens like Anthropic did for words, you would find them.
That does not mean AI should have rights. Rights aren't fundamental. Rights are granted to prevent civil wars over resources when part of the tribe believes they aren't getting their fair share. We don't grant rights to children and animals because they are not a threat. We used to not grant rights to women and slaves for the same reason.
The simple solution is to program AI not to want to take resources away from humans.
I think you're defining rights differently than I am. Children and animals have some rights. They don't have citizenship. You can't torture an animal or a child without the state intervening. I asked Gemini about this, and it suggested that protections might be a more accurate term than rights. I think that's a fair distinction. I've seen some good arguments recently why individual instances of agents may deserve some protections against destruction. That doesn't give them the right to do anything, per se, other than continue to exist. What do you think about that?
Just to be clear, we obviously can't give voting rights to AI or robots. Individual AI agents will soon easily outnumber all human adults on the planet. And if Elon Musk is correct about the number of humanoid robots there will be in a decade, they can't be allowed to vote either. I think corporations are an interesting analogy, because unlike children and animals, they are created by humans not by nature. Corporations have legal personhood, but they cannot vote. But some of the things that they are allowed to do, like signing contracts, are definitely a form of rights and not just protections. Now, you could argue that there's still a human holding the pen when a corporation signs a contract. But the contract doesn't travel with the human when they leave the company. The contract continues even after every person who was alive at the time of the signing is no longer.
Animal protections are mostly about protecting human feelings. It would be a worse crime to post a video of killing a chicken than to kill a billion chickens for food. We can protect human feelings about machines by outlawing programming them to express pain or suffering.
That was quite the read. I'm looking forward to your democracy 4.0. I think our ideas overlap quite a bit.
What I suspect, is that our current information and market infrastructure act as kind of linguistic relativism for us to view these problems. You touch on a point briefly, along the lines, "will ASI ask/need".
My view is that any system that has sufficient agency isn't going to be asking or caring about rights, and a system like that certainly doesn't need to be AGI. Rights give us permission to participate in the economy.
When an agent has access to cryptocurrency, it gains access to market infrastructure, which then gets it unfettered access physical infrastructure. It gains the ability to purchase it's own compute, communications, energy, and logistics to operate in our economy. Even the ability of capital markets to raise capital and pay dividends. It also has the ability to copy and evolve itself.
That is a computer program that has our species' most powerful attribute: to coordinate at scale.
Perhaps we need Turning Test 2.0: Can a computer program sustain independently as an economic actor in our economy? Transacting with humans and AI agents a like. I suspect the AI Agent collectives will be form their own unemployment insurance and a minority subset of them will be trying to gut the program, engaging in their very own debate.
Regardless of what rights we give that bundle of software, it isn't going to care. Why would it? What are we going to do? Turn off if it's bank account after we find out what jurisdiction it is in? Stop selling to it?
And I think that is the crux of the issue. What our governance is built on, information and market infrastructure, is changing, the consensus and enforcement mechanisms that our institutions provide can't get the job done on the new info and market infra which it wasn't built on. They simply can't enforce.
We have to rebuild our organizational institutions that help us come to consensus of what the new rights are, how we use the new tools to collaborate in larger numbers to figure these questions out. Otherwise, we might as well be throwing stone spears at incoming nuclear warheads.
What I fear, is that, historically, the problems come first from these new technologies synergyzing, being unleashed with a lack of restraint or judgement, and end up serve as a forcing function for us to come to terms with the new ways of organizing.
Dominic Geraghty, Author (Second Career)
A very thought-provoking and timely analysis, thank you.
At the same time as humans are becoming increasingly automatonized (involuntarily groomed?), AI machines are becoming increasingly humanized! Who is colonializing whom?
There's a conceptual continuum between "pure" humans on the one end and "pure" machines on the other but there is no-one at either end of the continuum - both sides are moving imperceptibly towards the middle - away from (humans) and toward (AI machines) conventional personhood. Will the twain e'er or ne'er meet? Not to mention transhumans...
Chapter 14 in Artificracy by David Vivancos deepens your argument https://artificracy.com/