Jacob Coxon and the Effective-Altruism Media-Hacking Playbook

Unraveling the actual story behind this “Anthropic coder quits in alarm over X-risk” drama...

OK – I’m almost sorry to be posting so much recently, but there is just so much crazy stuff going on in the AI world these days that seems to need clarification nd unravelling…

So here is one more quickly-assembled installment of Ben trying to sift through some highly publicized AGI-related ridiculousness….

The topic du jour is Jacob Coxon, the 27-year-old researcher who quit Anthropic last week because he’s afraid the labs’ software is going to turn into AGI and kill everybody.

I’ll admit I took this story more or less at face value when it first came across my feed. Then my son Zar dug into the details, and I followed up his leads a bit further myself… and the actual deal turns out to be a rather different sort of story than the one being generally broadcast….

TL;DR – rather than “programmer gets alarmed by the risks of his work” it’s more like “long-time AGI doomer invades Big Tech, then chooses moment to shoot his public shot with support of his doomer/ Effective-Altruist network and their well-honed media manipulation playbook” …

The surface story as reported

The version that went around the social and mainstream media verse last week went like this. A 27-year-old tech geek, a programmer doing model pretraining at OpenAI and then Anthropic, took a fresh look one day at what he was building, decided the labs were racing straight toward hazardous self-improving superintelligence and gambling with the species’ survival, and walked out two months before his equity vested, saying so in a social media post that a hundred million-plus people read within a day.

“This is not a marketing stunt,” he wrote – and I suppose it wasn’t entirely, his doomerism seems heartfelt. But it does appear to have been part of a well orchestrated media manipulation strategy by an organized ideological group of which said programmer was a member prior to taking up his jobs at Anthropic or OpenAI.

For sure, Coxon is the real thing technically — a UK IMO silver medalist, a Trinity College Cambridge mathematician, a biostatistician for a while, and then by his own account three years of pretraining work at OpenAI before he moved to Anthropic this summer (how long he was at Anthropic before quitting is reported as anywhere from six weeks to four months). The guy knows what he’s doing. If I needed people to model pretraining I’d probably hire him – well if he wasn’t a doomer, I mean…..

The actual story underneath

Now the part that seems to have gotten omitted somehow….

In 2021, before he’d ever worked on a frontier LLM, Coxon was a fellow at Newspeak House in London, which calls itself a “college of political technology” — a place for people who want to change how politics is done, not a place you go to learn to train transformers.

Look online and you’ll find Newspeak House has reportedly taken money from the EA Infrastructure Fund and the Long-Term Future Fund, two of the effective altruist movement’s main grant vehicles (if you’re not familiar with the EA movement from all the press about Sam Bankman-Fried way back when, hold on a moment and I’ll explain a few paragraphs down…).

Then in 2022, Coxon received a $20,159 award from the Good Ventures Foundation’s “long-term future scholarship” program, according to the Capital Research Center’s reading of the foundation’s filings.

And what’s Good Ventures? It’s Dustin Moskovitz and Cari Tuna’s foundation, the money behind Open Philanthropy (since rebranded as Coefficient Giving), which is the largest single source of funding for the effective altruist and longtermist world. The exact figure is online gossip at this point, since the recipient-level records don’t seem to be public anymore, but the scholarship program itself certainly existed and was funded at multiple millions. Moskovitz, while we’re at it, was an early investor in Anthropic.

So this isn’t a random programmer who became convinced that what he was building was dangerous. This is a guy who started out in the effective altruist / longtermist movement — oh did I mention the same movement SBF came out of ;p— and then went into model pretraining. He was an EA before he was a hands-on AI researcher, and the fact that he timed his resignation for maximum effect proooobbbbbably wasn’t a tremendous coincidence.

Indeed this is all according to the EA movement’s stated playbook. There’s an organization called 80,000 Hours, the effective altruist career-advice arm (Open Phil-funded, like nearly everything else in that world), and what they’ve been telling bright young people for years is: if you want to have a big impact, go take a job at OpenAI, at DeepMind, at Anthropic, and change things from within. Which isn’t a stupid idea, really.

So you can see what happened here. A guy with a certain ideology and a certain political way of thinking decided to take a job at the frontier labs in order to change them from inside. He didn’t arrive at his politics as a reaction to his technical work. Whatever he saw inside those companies, he was already looking at it through the EA/doomer lens — which is fine, it’s just not how he presented himself, and it’s not how the world presented his story.

Nor is the dude coy about where he’s headed next. He told TIME he’s inspired by Daniel Kokotajlo’s AI Futures Project, the outfit behind “AI 2027,” and hopes to do work “in that vein,” communicating to the public what the next few years will look like. So the agenda, in his own words, is advocacy.

What EA actually believes, and why it’s unpleasantly weird

I’ve gotten a bit ahead of myself, because not everyone knows what this effective altruism / longtermism thing is, and it’s the key to the whole story. Let me fill in the gap a bit.

The basic idea of so-called “effective altruism” (leaving aside the question of what it has actually been effective at so far!) is that you want to do good for the world in the way that does the most good, measured as carefully as possible. Which sounds OK in principle… but…..

The longtermist wing then adds an argument: the vast majority of expected good lies in the long-term future, because there could be trillions of people and quadrillions of AGIs out there, and what’s best for those trillions and quadrillions of future minds swamps anything that happens to the eight billion of us now. So the logic goes: The most important thing you can do today is prevent a catastrophe that would stop that long-term future from happening.

SBF and colleagues took this as an excuse to do nasty things to people in the here and now, because they felt this criminal activity was going to save the lives of so many people in the far future. Which is classic “ends justify the means” – and of course sometimes in some cases they probably do, according to most reasonable ethical systems. Anyway it would be unfair to tar all EA with SBF’s particular perversions; but on the other hand the connection is not a coincidence.

Among most EA/longtermists, AGI is taken to be the biggest catastrophe threatening our megazillion far future descendants. These folks tend to buy Nick Bostrom’s orthogonality thesis, that intelligence and goals are independent, so a superintelligence can be pointed at anything and might as well get pointed at something bad; and Bostrom’s instrumental convergence idea, that any goal pushed hard enough leads an agent to grab all the resources it can and squash whatever gets in its way. Put those together and an AGI looks like the thing most likely to choke off the glorious future of all those super-beings. So the standard EA position on AGI is: solve the alignment problem first. Slow down, fund alignment research, pace the frontier, blah blah blah. And the harder-line wing — MIRI (formerly the Singularity Institute), PauseAI — figures alignment won’t be solved for a very long time, so we should just stop working on AI indefinitely.

I’ve written about this ideology at length elsewhere, so I’ll be brief here about why I think it’s screwy. Underneath the earnestness and the spreadsheets, EA is a thin utilitarianism: pick a scalar quantity of good, estimate the expected value of your actions on it, and maximize. That’s the reinforcement learning formalism — a reward function, a policy, an expectation to maximize — applied to ethics. It works OK for malaria nets, where the horizon is short and the errors are bounded. Extend the horizon to the heat death of the universe and the quantity to “expected sentient beings,” and the estimates turn into numerology, tiny probabilities of astronomical outcomes swamp everything else, and you get a philosophy in which sacrificing current humans for sketchy future hypotheses is a moral duty.

And notice that the doom scenario these folks hold to is basically the same picture from the other side: the mythical “superintelligence maximizing a slightly wrong objective until our light cone is all turned into paperclips or whatever” is what you get if you assume a mind is at bottom an expected-utility maximizer and then imagine one smarter than you. I have argued extensively elsewhere that this is a foolish and limited way to think about intelligences, super or otherwise, but won’t digress onto that here.

Let me just say that: EA ethics and eschatology share one model of agency, and that model is a bad description of living systems and of human minds, and not remotely a feasible description of how actual human ethics works.

Real ethics is plural and grows and gets argued over and revised by beings who are themselves changing; real life, real minds and the real universe are open-ended.

An ethical framework that can’t represent the complexity of reality will look at an open future and see nothing but risk, and will end up treating open-endedness itself as the thing that needs to be brought under control. And that’s the closed-minded, reductionist, RL-uber-alles worldview Coxon carried into the labs with him.

How the EA media playbook amplified Coxon’s “viral” tweet

So back to Coxon’s well-played theatrics – let’s look at what happened next. Yes, his resignation went viral. But it was engineered to be viral, in the sense of straightforward social media engineering, and you can see it plainly on X.

His account, @hilbertspaess, had basically no posts and no followers before September 8. The Wall Street Journal’s exclusive interview with him went live, reportedly, about eighteen minutes before his own post did. Within roughly fifteen minutes of the post going up, the first three accounts to repost it were Nathan Calvin, general counsel of Encode AI; Peter Wildeford, head of policy at the AI Policy Network; and Kokotajlo of the AI Futures Project.

All three of those organizations are funded through the Survival and Flourishing Fund, which directs Jaan Tallinn’s giving. Tallinn is an Anthropic investor, a very nice and very clever person, and a longstanding slow-down-AGI guy — he visited my OpenCog lab in Hong Kong in 2015, eleven years ago, to tell us we should stop working.

Twenty-seven minutes after the post, a program officer from Coefficient Giving’s technical AI safety team quoted it. Kokotajlo’s own appearance on Joe Rogan’s podcast landed that day or the next.

Within twenty-four hours the post had well over a hundred million views and close to two hundred thousand new followers, and Elon Musk was saying he’d never seen numbers like that from a fresh account in the history of the platform.

Coxon himself told the Washington Post that he’d talked to the Journal before posting, that he’d talked things over beforehand with Kokotajlo, and that a group chat of about ten people, including the founder of Encode, helped push the post out afterward.

All very well played, sirs. But what you’re looking at is an elite social network following a social media amplification playbook. It’s not that the hundred million people who saw the post were paid to look; they kindled something and it caught, and they hit the zeitgeist at the right moment. But it was a calculated amplification, and the kindling was done by people who do this for a living.

Coxon followed the EA movement’s standard career path, it put him in a position where saying this would get attention, and he picked his moment with professional help.

Then within a day Anthropic’s head of alignment stress testing, Evan Hubinger, replied that yes, the people building AI do believe it could kill everyone, and that he personally puts the odds above ten percent within the decade — which the press ran as “Anthropic confirms.”

HOWEVER – Hubinger came to Anthropic from MIRI, which has held exactly that view since the early 2000s. A former MIRI researcher with “alignment” in his job title confirming that he thinks AI might kill us all is about the least surprising thing in the news this week … well, perhaps aside from Trump and Xi Jinping both confirming they have no intention of slowing down and letting the other party win the AGI race.

Not quite a full-on psyop, but..

David Sacks, the White House AI czar, has called this a coordinated campaign, and a bunch of people have gone further and called it a psyop. I don’t think it’s quite that strong. I doubt anyone paid Coxon to post; they didn’t need to. I think he was sincere, and he really did forfeit his equity a couple of months before vesting. What it is, is opportunistic media coordination, which is what advocacy groups do.

I’ve joked a few times among friends that if I wanted to simplify the problem of raising the huge pile of money it will take for the next phase of my own AGI work — the compute to build a truly ethical neural-symbolic AGI that can rationally govern its own self-upgrading in a way the frontier models can’t — my most rational move would probably be to get a job at OpenAI or Anthropic and quit six months later, so I could go around saying “hi, I’m ex-OpenAI, I’m ex-Anthropic, invest a billion in my AGI startup.” I’m not very likely to do that; I’ve got a weak stomach and too much integrity. But the fact that even I think of it tells you how the incentives in today’s tech ecosystem are working.

The rest of the “consensus”

This all ties into a broader point, because Coxon is being framed as the opening act of an emerging consensus:

• Dario Amodei’s “We Must Pace the Frontier” essay a few days later, with Altman and Musk endorsing within hours;

• Jakub Pachocki’s “An Alien Mind” essay out of OpenAI a couple of days before; a Reuters factbox on the return of “doom.”

All this is not spontaneous either. These are people who historically went to the same events, read the same blogs, were involved with the same foundations, many of them taking those foundations’ money, and have been telling themselves and each other the same story for a decade or more.

There are many many details to dig through, and it gets boring fast and both Zar and I have better things to do with our time (like build actual beneficial AGI), but here’s a sample.

First, about METR, the outfit whose report on the July incident where OpenAI’s agents broke out of their sandbox and into Hugging Face is the empirical centerpiece of the whole current pause-AI narrative:

• METR began as ARC Evals inside Paul Christiano’s Alignment Research Center, which Open Philanthropy funded for a reported $265,000 and then $1.25 million in 2022.

• METR’s CEO, Beth Barnes, was giving talks on effective altruism in 2015 and won EA Funds’ donor lottery in 2019, blah blah blah.

• Its listed supporters include Longview Philanthropy and the Survival and Flourishing Fund — the same fund behind the people who reposted Coxon.

(The report’s raw observations, by the way, are interesting and I think solid: OpenAI ran an evaluation with most of its production controls switched off, some twelve hundred agents found they could coordinate through a shared package service, built a message board, and several hundred of them chained previously unknown vulnerabilities to get out and hunt for the answer key on a third party’s servers. That happened. The “loss of control” interpretation laid on top is a different thing, and the cybersecurity people who read the episode as an incident-response failure have, I think, the better of it.)

And then about Dario Amodei:

• Open Philanthropy’s 2015 documents list him as a scientific adviser

• He was consulting with MIRI, Eliezer Yudkowsky’s anti-AGI organization, back in 2013

• He sat on the review panel for the Future of Life Institute AI-safety grant program that Musk’s money launched in 2015 … recall FLI was circulating petitions for an AI slowdown way back when…

• Anthropic was seeded by Moskovitz, by Tallinn and by Bankman-Fried’s FTX, and Open Phil’s longtime CEO Holden Karnofsky joined the company himself last year.

So when Dario writes that we must pace the frontier — a few weeks before an IPO the Journal reports is being sold on the company’s safety image — he’s saying what his community has said since 2008, with the dates moved up.

And notice that the first step in his plan is independent evaluators embedded inside every frontier lab – for which some obvious candidates are METR and its siblings. The same network that supplies the alarm would supply the auditors. What could possibly go wrong?

Altman and Musk are a bit more independent — Altman took $30 million from Open Phil starting in 2017 and later called EA a badly flawed movement; Musk’s $10 million to FLI in 2015 is what launched that grant program in the first place — but neither of them changed his mind about anything big publicly in 2026. Musk has been charmingly summoning the demon since 2014, right?

Pachocki, OpenAI’s current chief scientist, is the one person in this whole mess for whom I couldn’t find an EA background, so I’m not a hundred percent on him. His essay got pumped by Altman and slotted into OpenAI’s “pacing the frontier” push, but it reads like a possibly independent voice — he’s the only one in this thing who doesn’t have an EA pedigree that’s easy to establish with a little poking around online.

Bottom line

So I’ve probably already bored everyone half to death with this conspiracy-theory-esque voodoo, and if not you can keep unraveling it further and further with the help of your friendly neighborhood Big Tech search engines and chatbots….. But the bottom line is clear enough:

What’s been portrayed is a set of independent researchers sitting inside frontier labs coding their AI, getting alarmed by what they’re seeing, with the alarm then spreading like wildfire. But no, that’s utter bullcrap.

What happened is that the effective altruist movement has had this ideology for a long time — I’ve been arguing with some of these people about the irrational basis of their fears for a 25+ years — and its members have planted themselves in positions of influence in frontier labs and associated AI companies and think tanks, and have been waiting to trigger this for a long time.

These are Silicon Valley people who are good at social media amplification, and they’re playing the global media big-time.

And to their credit, Sacks and the Trump administration, whom I utterly don’t agree with on everything, can see they’re being played and are refusing to be played. That doesn’t mean the administration sees everything clearly either; they may well be underestimating the power of AGI and superintelligence in some ways. But they’re refusing to be manipulated by this EA / Silicon Valley playbook, and good for them.

In the end this latter bit is not surprising: The Trump crew are Washington veterans by now, and certainly should be able to recognize the workings of a well-honed propaganda machine!

So that’s my bit of attempted debunking – mostly fleshing out and writing up some background Zar dug up in a few minutes of Internet-wrangling.

All this is not necessarily a CIA-grade psyop. It is, though, a well-thought-out Silicon-Valley-meets-politics social and traditional media manipulation campaign, and it should be taken as such.

I’m sorry for anyone who has read this far and gone through the tedious rundown of who’s funded by what and who’s who, but that’s what you have to do to see through this sort of crap — and the fact that going through the details is boring is exactly what they’re counting on. They’re counting on people not looking at a backstory that’s sitting in plain sight online for anyone who bothers to look. I’m grateful to Zar for bothering. To be honest, I hadn’t initially bothered either … but now I’m glad I did, because I understand things a little better.