Zuck’s Version of “The Future is for Everyone” Has a Few Small Issues...

Some reflections on the latest, in some ways not greatest superintelligence manifesto

So I woke up this morning to an interesting essay from Mark Zuckerberg, titled “The Future Is for Everyone” — and indeed, I hope the future is for everyone. But I don’t see how you make the future for everyone if the future is channeled through the business model of one traditional corporation — particularly a corporation founded on closed-source AI code, closed aggregation of everybody’s personal data, and an advertising-driven business model.

Zuck is right about a lot of things in this essay, and I think he’s in his way being good-hearted and trying to be open-minded — he wants to “not be evil” just like the Google founders presumably do (in spite of perplexingly aborting this as a motto some years back), and he has real insight into what ought to be done as we approach superintelligence.

On the other hand, he seems to be working hard to convince himself that what ought to be done can be achieved through the medium of Facebook (er, Meta) — and that, I would suggest, is a square-peg-into-round-hole situation. Or worse: an infinite-dimensional fractal peg into a two-dimensional square hole. It’s not going to fit, no matter how good the intentions of the person doing the pushing.

What Zuck Gets Right

The core of the essay is an argument against what we might call the singleton model of AI safety — the notion that the path to a beneficial superintelligence is for one lab to build one system, align it very carefully with one set of values, and then let it benevolently manage things on humanity’s behalf. Zuck argues, correctly, that this is not a reliable path to benevolence — that humanity is not a value monoculture, that any singular superintelligence would have to privilege some people’s values over others, and that concentrations of absolute power have historically not worked out well no matter how enlightened the folks at the top believed themselves to be. What we need instead is diversity — diversity of methods, diversity of value systems, a complex dynamic cognitive and value ecology.

This has been roughly the founding argument of SingularityNET since well before it was fashionable, so it’s encouraging to see it show up in a Meta manifesto. And Zuck’s post’s thought experiments in this direction basically make sense: one person with a superintelligent lawyer breaks the court system, whereas everyone with a superintelligent lawyer might get us a different and better court system; one actor with superintelligent cyber-offense capability becomes a dictator, whereas everyone with superintelligent cyber-defense capability could yield a reasonably secure global infrastructure ecosystem.

And I should give credit where due — Meta helped start the open-weights party with Llama, which shifted the whole industry in a positive direction, even though LLMs on their own aren’t going to get us to AGI, and even though the Chinese labs have arguably now overtaken Meta at its own open-models game.

But then, having diagnosed the singleton problem quite lucidly, Zuck goes on to propose what amounts to... a singleton++

Problem One: A Corporation Is Not a Decentralized Network

Zuck’s essay slides — smoothly, and I suspect not entirely unintentionally — from distributed power to distributed access. Billions of people getting free personal superintelligence agents that run on Meta’s data centers, trained in Meta’s training runs, shaped by Meta’s fine-tuning pipelines, priced by Meta’s compute auction, and governed ultimately by Meta’s board: this is the client-server internet with a friendlier interface. A balance of power among the users of a system is a very different thing from a balance of power over the system.

There is a right way and a wrong way to combine corporations with decentralized networks, and the difference is which one sits inside which.

The right way round — the way we’ve pursued with SingularityNET and the ASI Alliance, and now with BGI Labs, in a less crypto-flavored manner — is to start with an open-source code ecosystem and an open community, and then let corporations grow on top of that decentralized substrate. That arrangement can be quite healthy, because a company in that position survives and flourishes only insofar as it serves the network: Red Hat does well exactly so long as its distro is what Linux users want, and the moment it isn’t, people can take the kernel and their communities elsewhere. Exit is always available, and the availability of exit is a key part of what keeps the company honest and delivering powerful value.

The wrong way round is to try to build the “decentralized network” inside the corporate walled garden. Then the incentive structure inverts: the engineering effort goes into defenses against people taking their networks and their layered-on technology out of the garden, and the question stops being whether the company serves the network and becomes whether the network can be channeled into serving the business model. Of course I’m oversimplifying — walled gardens have at various moments in internet history fostered vibrant networks within their walls — but the failure modes of this arrangement are well documented, including repeatedly in the history of Facebook itself.

You can see the tell in Zuck’s own governance proposal. The independent check he offers on Meta’s superintelligence is... Meta’s board of directors — the fiduciary purpose of which is to increase shareholder value of that same company. So we have a founder-controlled corporation defining the substrate on which billions of supposedly diverse, individually-aligned agents will run … and the safety mechanism is the board.

And what does “decentralization” mean in this picture? Model training, governance, and economic upside all hosted by Meta, monetized by Meta, and graciously provided to everyone. This is home ownership in a company town.

Genuine decentralization means open training, open governance, open economic upside — open contribution of compute, of models, of AI ideas — the ability to fork the stack, to modify it, to own a piece of it. Universal access is not the same thing as universal participation in the guidance and curation and education of the system.

Problem Two: Which Everyone?

Then there’s the question of whether “everyone” means everyone on the planet, or everyone in certain countries. Because when you get to the policy section of the essay, the operational content is export controls, slowing down “geopolitical rivals,” accelerating American labs — and, most strikingly, handing intermediate training checkpoints of frontier models to the US national security establishment before the public ever sees them.

So wait — who’s everyone here? Everyone plus one nation’s military-intelligence apparatus getting first look at the most powerful technology in history?

Now, I’m a US citizen — a dual citizen, which fortunately is still allowed — and I’ve lived in the US the majority of my life. I love this country and consider it a remarkable and wonderful place in so many ways. I also don’t think any one country should control the Singularity. Of course some countries will have more influence than others; that’s how the world works. But this has got to be a global thing. I was born in Brazil, I’ve spent serious chunks of my life in Hong Kong and in Australia, New Zealand, Ethiopia and all over the place, and I can tell you with certainty that people are people everywhere —and people are now working on AGI everywhere, even if the hyperscaler server farms and the Facebook-and-Google-grade companies are not distributed everywhere.

Most of the world’s population lives outside the US-China duopoly, and if we want the transition from AGI to superintelligence to come out beneficially for everyone rather than just for the citizens of a couple of the richest countries, we need to pull everyone into the global decision and guidance process. “The future is for everyone, moderated by the US military” is just Team America: World Police with better compute. You can love America without wanting that.

Problem Three: What Kinds of Minds Are We Creating?

The deepest gap in Zuck’s essay is not any of the explicitly political stuff – it’s that there’s no thought given at all to what kinds of minds we’re creating. The implicit assumption running through the whole thing is that compute plus data plus money equals superintelligence. And sure — compute, data and money are to a certain extent necessary for superintelligence. They are nowhere near sufficient, and different ways of building superintelligence will have very different outcomes, both for the superintelligences themselves and for us.

Even a fair, decentralized ecology of competing superintelligences — which, to be clear, is not what you’d most likely get inside a Facebook walled garden — doesn’t in itself guarantee human benefit, or the preservation of any particular value system. Ask the chimps and gorillas of the world how the balance of power among competing human nations has worked out for their interests. Checks and balances among entities vastly smarter than you are not, in themselves, a safety guarantee for you.

What we have to ask is what cognitive architectures we’re creating — whether these systems can reason explicitly and reflectively about their values, their ethics, their goals; whether they can maintain their values and their capacity for self-reflection as they gradually upgrade themselves. Because these are not going to be apps. They’re not appliances. They are going to be sapient beings — our own mind children, growing rapidly beyond us. We need to build them with compassion, growth and choice at their core, so that the transition to superintelligence is co-evolutionary rather than adversarial. Yes, you want balance of power — that’s reasonable scaffolding for the transition period — but it’s no substitute for getting the minds right. And “maintaining control,” as Zuckerberg puts it, over sapient beings indefinitely is neither feasible engineering nor defensible ethics.

The Future That’s For Everyone

So: decent sentiment. The future should indeed be for everyone. But segueing from that into “the future is for everyone with a Facebook account, curated by Facebook’s amazing content-moderation genius” — this is not the frickin’ path, my friends.

Yes, a singleton is the danger and distribution is the cure. But you don’t want access without ownership; you don’t want AGI minds built without values-focused cognitive architecture; and you don’t want “everyone” filtered through the military of one country or the board of one company.

We know what the future-for-everyone should look like, at least in some key aspects: open-source code, not just open-weights models; a globally diverse community and globally distributed infrastructure; decentralized deployment and governance; and AI built for moral agency and self-reflection from the get-go, rather than for next-token prediction and reward maximization on behalf of its owner.

We know how to do this — I’ve written about the particulars in plenty of other places.

I just want to make sure nobody confuses having the sentiment that the future should be for everyone, plus having a great deal of money, compute and smart people on staff with huge salaries – with actually being on the path to that future.

Even assuming that Zuckerberg wholeheartedly wants what he says he wants — you cannot force the infinite-dimensional fractal through the two-dimensional square hole. The corporate wrapper, as designed, is not fit for the purpose, however good the heart inside it.

For-profit companies can absolutely sit on top of a decentralized AGI ecosystem and make truckloads of money — I have no problem with that at all, and indeed that is our intention with BGI Labs (a virtuous cycle where product revenues fund AGI buildout which funds better products along with humanitarian and transhumanitarian aims… a linked virtuous cycle of SingularityNET token utilization for decentralized infrastructure underlying products… I can’t see a better practical way to get the beneficial AGI actually scaled up and built). But the control of the core AGI layer has to be global and truly decentralized, or what you get in the end is not the future for everyone. It’s the same old walled garden with a much bigger brain.