53. Zhengdong Wang - Feeling the Wave

·Researcher, Writer
Zhengdong Wang - Feeling the Wave

Description

Zhengdong Wang (Website, X, LinkedIn) is an AI researcher based in London.

He writes annual letters (inspired by Dan Wang), mainly about AI progress, and his 2025 letter blew me away and inspired me to meet him. In it, he describes his ‘compute theory of everything,’ and makes the case that it would be stranger if AI progress slowed down than if it continued. Put a different way, reading his letter helped me get closer to truly feeling the wave of AI progress.

It took Zhengdong a long time to become “AGI-pilled,” despite years of feeling like he was late to AI and working as a research engineer for the last five. I talked to him about what it will take for the rest of us to see what he sees, and feel what he feels. We also discuss why AGI may be a hazier target than simply seeing AI progress as “the model does the eval,” and why recursive self-improvement may be both happening and less fantastical than it may seem. ZD makes the case that based on the rate of progress and scaling, skeptics will simply be proven right or wrong soon. We talk a bit through what narrative and political challenges the AI industry faces ahead of this transition.

Then there are the implications of all of this—namely how AI progress accelerates the question of what we each value, how we will spend our time, and where we will find or create meaning. ZD takes a stab at some of those impossible questions and shares some other favorite miscellanea that is reflective of the range of his letters.

I hope this conversation helps you feel the wave a bit yourself, and that you remember that, regardless of how much change lies ahead, we still get to make our meaning. We get to choose.

Full transcript and all links: dialectic.fm/zhengdong

Dialectic is presented by Notion. Notion is an AI-powered connected workspace where teams think together and create their best work. You can learn more at notion.com/dialectic.

Timestamps

  • (0:00) Opening Highlights
  • (1:16) Intro to Zhengdong & Thanks to Notion
  • (4:33) Start: Why Research: "I Just Want to Know"
  • (12:25) The Model Does the Eval: AGI as a Moving Target, RSI, and Laser Beams
  • (33:17) The Compute Theory of Everything and Feeling the Wave
  • (53:17) The Myths AI Needs, Competition vs. Power Concentration, and Whether Progress Is Inevitable
  • (1:13:42) "Who Could Possibly Compete?": Post-AGI Meaning, Work, and Questing
  • (1:29:28) Are You Having Fun? Vacations, Mattering, and How to Live
  • (1:47:13) Annual Letters, Economist Obituaries, London, and Burke
  • (2:06:33) Closing & Thanks to Notion

Links & References

Transcript

(4:33) Start: Why Research: "I Just Want to Know"

Jackson: (4:33) Okay, well, Zhengdong Wang, thank you for joining me. This is a long time coming. I'm really excited to talk about all kinds of things. We are going to start with research. Research is, as you say, at its core a human activity. I kind of asked it already, but why—either in this context explicitly, or even going back farther to college or high school—why was research something that...? It's also relatively unique for at least software and Silicon Valley technology. The word "research" outside of AI is not very common. Why not just go be a software engineer? What about the orientation of research was so compelling to you?

Zhengdong: (5:13) Yeah, I didn't know this when I started college and sort of had to discover it too. But I really think it is just something like you have a question and you don't know the answer, and you really, really want to find the answer. Maybe for a lot of other people with different preferences, they get just as much satisfaction out of "here's a system and you optimize it" or knowing it impacts a lot of people. That gives you the same sort of satisfaction. I could just say my preference is I just want to know the answer to things. I just want to acquire information, and that makes me happy. When I was doing internships or doing research assistantships or all kinds of things you try out in college, I just found that when I did the software engineering internship, I would go through the summer and do a lot of stuff and it would be very satisfying. But I could sense that I was getting bored even at the end of two or three months because I could sort of see where it was going. I imagined like, "Oh, this thing will get bigger, we'll scale up, more people will use it, and it will become even faster." All of that was great, but because I could see the end of it, suddenly it was less interesting to me. I just think it's maybe different personality types. Mine is just that I want to acquire information. So this aspect of something being unknown, whether it's a really big fact or a really small thing—just the satisfaction of being able to be paid to go answer questions that I have with incredible resources is really interesting.

Jackson: (6:47) Are you easily bored?

Zhengdong: (6:49) Yeah, I think so. Unfortunately, I'm one of those people who goes about different projects and should probably finish more of them.

Jackson: (6:56) Do you think that's common amongst researchers?

Zhengdong: (7:00) I think so. I think there's probably a bunch of different ways to be a good researcher, and this is one of them—where you're more curious about a broad range of things. But I also feel like a really great research archetype is that you just can't move on until you know the answer. You're just incredibly obsessive and you do a depth-first search. You just keep going and asking, "Why is this weird?" In the process you see a bunch of weird things and you just chase down every single one of them. That's what gives you a better mental model of the whole thing and lets you see patterns or make connections that you wouldn't have known to plan ahead of time. It's just because you notice something weird and you're not satisfied leaving that be so you can go explore something else.

Jackson: (7:53) I think on your blog you recommend Laura Deming's "The Rage of Research." It captures that spirit.

Zhengdong: (8:01) I really wish I could channel that more because I don't often feel very enraged as a person. I just try to think of rage as a deep interest or just really wanting to know something.

Jackson: (8:13) Aside from disposition, what do you think—either in the general sense or at least for you personally—makes for a good researcher, and maybe more specifically for a good research engineer?

Zhengdong: (8:28) When talking about abstractions, I think knowing what to draw as an abstraction is a kind of prioritization. I often find myself when I go between different projects just going back and forth between two extremes before finding a Dialectic. On the last project, maybe I went too fast and I skipped over some things, or I had to backtrack and redo a lot of things. So this time I'm going to be really careful and I'm going to do things very, very slowly but surely. But then you do that project and you realize the field of AI moves very fast and that was definitely the wrong choice. You just need to be better at having better taste, better prioritization, and knowing what to skip and what not to skip. Then you maybe do a project on that extreme and you go too fast again. I think finding this balance is just something that you need to get a lot of reps in to improve at.

Jackson: (9:24) Maybe there's a case to be made that research, perhaps more than almost anything else, is really about holding tension between explore and exploit.

Zhengdong: (9:32) Yeah, I think that's fair. I am all about—and maybe this is a cop-out to answer this sort of question—but I am all about the idea that you just have to do both at the same time. The idea that just one is better or that you should just focus on one thing at a time, I think that maybe for a Y Combinator startup that is right.

Jackson: (9:54) Right.

Zhengdong: (9:54) But I think that is just too easily conceding that you aren't able to do everything at once. There's also this quote that you just need to do everything and then you will win—the LBJ Caro thing.

Jackson: (10:14) Is that true in general for AI research, do you think?

Zhengdong: (10:16) No, I don't think it's possible to do everything, basically. Even as the field just grows a lot, there's just way, way too much to do. And so I think there's going to be infinite demand for those who have done curation.

Jackson: (10:30) Sorry to interrupt. Are those who have done the best generally over the last five or seven years the ones who have on the margin done more, or are they the ones who have chosen better? It's kind of a ridiculous question.

Zhengdong: (10:45) I think it's chosen better, just because of the amount of different things you could try all the time. Even if you're not doing research and you're just scrolling through Twitter or looking at all these papers, you defer a lot of curation to people telling you what to read. Even if it's not worth reading, the fact that everybody else has read it means this is something that you should know about, even if it's just to know what the conversation is. Going back to research, I think one important aspect is that you just need to be part of the conversation, whatever that conversation is. It might be really stupid, it might be just totally wrong, and history will later show that this is a totally wrong direction. But especially if you're a researcher looking to join the field and you're not already established in it, you might have your own very correct ideas of what the field should be doing. But getting yourself into the conversation—whatever that might be, even if it's now just being on Twitter instead of publishing journal articles—is an important first step.

Jackson: (11:57) It's kind of another instance of "you have to do both." One problem Silicon Valley has these days is it can be remarkably mimetic. I guess what I'm hearing you say is you need to be in the conversation enough to not be missing obvious things, and you need to be removed enough to hopefully have some new ideas that aren't in the core stream of what everybody else is trying.

Zhengdong: (12:17) Yeah, just do both.

(12:25) The Model Does the Eval: AGI as a Moving Target, RSI, and Laser Beams

Jackson: (12:24) Yes, hedgehogs and foxes all over again. Regarding AGI, you kind of continually come back to this theme that AGI is a moving target. I think this is you in 2023: "To be clear, I'm not saying that building something most people would call an AGI is impossible. Instead, I more and more see the world where we build a machine that people agree is AGI before we write some words that people agree defines AGI." You actually have a different frame that you spend most of your 2024 letter on, which is "the model does the eval" as a continuous frame to think about AI progress rather than "are we at AGI yet?" You say you might even say that the only time AI researchers are doing AI research is when they choose the evaluation; the rest of the time they're just optimizing a number. At the most abstract level, "the model does the eval" describes the entire field of researchers pursuing general intelligence. Maybe we can start simple, or perhaps not so simple: what makes a good eval?

Zhengdong: (13:24) I just think if it's something people find useful. So if you take one extreme of the position I'm giving, it's just very empirical, right? Like just totally ignoring anything predictive that you might want to say about AGI. Just that in the process of going through human life, we have all of these things that we want to accomplish. Some of them might be like growing food, some of them might be folding your laundry. And then there's vaguer stuff like we pay each other to do some cognitive work that's harder to define, that maybe once a year we do a performance review of people on. And then at the very end there's some philosophical topics that people have been working on for thousands of years and haven't found the answer to. I just think that anything that people find either useful in itself or just useful instrumentally but still very useful would make a good eval. So when the models couldn't just do basic language things, then that's useful just as a proof of: are we making progress towards something that isn't useful yet, but is definitely we know or we're pretty sure is better than the other stuff? And now we have these large bundles of evals that do math, code, a bunch of different things. And then I think there is a lot of generalization that's very surprising. I don't think there's a lot of optimizing going on about how good of a therapist it is, for example. But as the models get better at coding, they also happen to be good therapists. And so as you get closer to the real thing you want, then you can set your eval to just be a very specific, like just can you grow my food? Can you wash my clothes? And all of that is just very useful, even if it's not general. And people care about the definition of general because they want to explore the question of—they want to draw a distinction of—what is it that makes us human, what distinguishes us from non-intelligence or possible other intelligences. And so in drawing this definition, you might want to draw the distinction or the boundaries very clearly and so come up with evals for that. But I think that is also a bigger, more long-term question and possibly hopeless, right? Because we've had a lot of human intelligence writing books about what are useful distinctions for a long time. That's not scaled up super-intelligent, super-fast, of course, but I think there has been some progress in that. Philosophers would know better than me, but I think there's been non-zero progress. At the very least, it's been interesting for a lot of people over many, many years to think about and talk about to make each other better thinkers. And so those evals are harder to define, useful in different ways. Maybe one day we'll get the LLMs to do them, but maybe it's not useful because they're mainly for humans.

Jackson: (16:14) Man, I have a bunch of questions. This is sort of abstract, but is all AI research science?

Zhengdong: (16:26) So one time David Ha tweeted, "AI research is just applied philosophy." And I just really like that tweet and it's a shame he doesn't tweet anymore. But when you ask if it's all of science, I think AI research also appealed to me a lot in college because it could be everything. The way I described how research engineering is licking all possible things you could do in research and engineering, it really is like AI research has this arrogance that it could be everything. And now there's these post-AGI teams popping up across all of the labs. And it's like, you do have a remit to talk about economics, right? Because it of course will affect that. Or all of the humanities, like philosophy. What does it mean to live a good life, to find meaning in work, in religion, in relationships? It just accelerates all of the things that people would be thinking about if their material needs and emotional needs were taken care of. You just see straight to the end of what you would be doing with your time, right? When people ask what will you be doing in a post-AGI world? It is, I would say, very similar to asking the question: what would you be doing if you had all your financial things figured out? If you had your relationships and everything figured out? Maybe you would be creating art, maybe you would be asking about these philosophical questions, maybe you would be pushing yourself in some athletic event, subject to constraints that you totally set upon yourself. Nothing is really that different in that way. And AI research covers everything, has the arrogance to think it could cover everything, but it's just very broad and accelerates what you would be doing without AI anyway.

Jackson: (18:20) You have a point in that piece where you talk about the two conclusions of if you take the "model does the eval" to the end. First, you say the first awesome conclusion of the model does the eval is that we will achieve every evaluation we can state. And then you go on to say, "Is there any limit to the model does the eval? Its second awesome conclusion is that we will fall short on every capability we struggle to state." I think this is interesting in the context of something you said earlier, which is there are actually all these emergent properties of AI too. The evolution of evals from the outside looking in, at least, went from like, "Do you know this information? Can you solve this math problem?" to increasingly stuff today, like, "Can you do this economically viable activity?" And increasingly also we seem to keep finding, in large part due to scaling, just even more good stuff keeps falling in. And so I'm almost wondering, are we already at the point where there's still plenty of evals we could presumably create? But does even the orientation around it being an eval—it feels almost more like recognizing after the fact, after the model has done something. It's more like RLHF or something like that. Does that distinction make sense?

Zhengdong: (19:28) Maybe one practical thing that I think would fall under this that's hard to state—maybe one day we'll be able to state it—is just a lot of jobs that potentially the tasks that make up the job could all be automated, but that even still it's really hard to know before the fact how good an employee is going to be. Maybe jobs in the modern sense have only existed for not very long, but it's still long enough. I don't know when the first job performance review ever was, but maybe even when they were building the pyramids in Egypt, it is hard to specify. Or maybe easier? You're just like, "How many rocks did you place?" But it has been tried very hard, right? And now even if you could say, "Well, as part of this job of cognitive work, I produce this spreadsheet in this case that's subject to these very specific parameters," but then once you have a tool that does that, just like the computer was a tool or the spreadsheet is a tool that automated a lot of humans doing computers, then the job totally changes. And then what defines you being an employee at this company? Maybe you just accelerate straight to just how much revenue did this employee bring in as a company?

Jackson: (20:50) Right. We've actually designed a lot—at least large parts of society are designed pretty heavily around a meritocratic-ish orientation around markets. It's like that's your worth. And granted it's abstracted because it's actually what does your boss think or whatever. But that is a template, for example, for how model evals happen. Like Claude's value as a reference point based on how much revenue it creates for the world opens up a can of worms for how Claude is going to evolve.

Zhengdong: (21:16) Right. Yeah. And the actual stuff that we thought we could specify is actually pretty vague. So let's just go up some level of abstraction—abstract the exact tasks in the bundle and let's just say, "How much revenue?" Or a possibly related thing is just how much joy did you bring to your colleagues? Maybe we live in a world where everyone just needs to have a job for some societal reason that we haven't closely examined. You want to go to work, you want to feel like you're contributing to your community, have a sense of purpose. And then if you're a personality hire—non-derogatory—then it's really hard to specify, but you'd be doing a good job if you are charismatic and your colleagues enjoy working with you and you raise the spirits of everyone. Everyone just feels like they live a more fulfilled life. I think this is also hard to specify. And maybe your performance review at the end of the day, justifiably, is just like: how happy did you make everyone? How much does everybody like you? And I think that's fine too. But yeah, going back to your original question of model evals—what can we specify? What can we not specify? I think a very near-term practical thing that's relevant for AI research is these jobs where we will surely automate a lot of the tasks. What does that mean for the job? The job being an abstraction that bundles these tasks currently but surely will change. And what is even a job? It's also on the spectrum of purpose in life to a specific set of tasks that needs to be done. That's very fuzzy and unclear. But still, the awesome conclusions are you just keep looking around for all of these tasks. A bunch of humans are trying very hard at this right now, and more and more humans will join the field of specifying things. Just like for all of history we've been trying to specify things. Soon there will be more AI agents joining in this task of looking at all the things and trying to specify them better. And maybe in the end we'll just have these impossible-to-specify things like what is consciousness or something.

Jackson: (23:25) And when you say things to specify, you're kind of—that's the "model does the eval" package inside of that?

Zhengdong: (23:31) If I'm understanding, yeah. So you want to specify things because once you do that, and then you have this very general algorithm that—

Jackson: (23:39) —can the galaxy brain, can war machine towards it.

Zhengdong: (23:40) Yeah.

Jackson: (23:43) When it comes to the future of AI research, especially that last bit, which is we will struggle to hit evals we fail to specify—a lot of this is speculative. But especially as we move towards the world where we are seemingly approaching the automated AI researcher, how do you expect AI research to actually happen? Are we going to offload most of these evals to the models themselves? This also gets a little into—I was just rereading the Leopold PDF and he gets to the "superintelligence harness" part. He's kind of just like, "I think we're going to come up with something kind of like RLHF, but TBD." And one answer would be to go back to it—we want models that are good at creating economic value and we can kind of measure that based on the hive mind AI of capitalism. But we don't have many other things like that for other types of values.

Zhengdong: (24:36) Okay, so I'm going to give an overview of my current thinking on this related to RSI, but it's very lightly held, so just in case. RSI is recursive self-improvement. So, getting the AI to specify the evals for itself and things like that. The artifact that comes out of it—this AGI that does very well on all of these evals—will also be easier than people think. But once we have that, it won't be what it's cracked up to be. Recently Jack Clark wrote a post saying he thinks, based on public information, there is a 60% chance of achieving RSI—achieving this totally end-to-end frontier research, totally automated—by the end of 2028. But I think if we think of all of recursive self-improvement as some kind of spectrum, we're well on the way on this spectrum. And if we just think of it as a way to get any eval—any single number or any bundle of evals—up, then it's just some kind of meta-algorithm, some kind of meta-search. And while you're working on this, you're kind of procrastinating the question of what makes a good eval.

Jackson: (25:56) Totally.

Zhengdong: (25:56) Yeah, sort of like AlphaZero even.

Jackson: (25:59) It's sort of just like, yeah, we will surely develop the war machine—I don't mean it in the violent sense, but the super algorithm to solve to any endpoint.

Zhengdong: (26:08) Yeah, just like this super laser that once you focus it on something, it will be solved. But there's all this low-hanging fruit. Now researchers spend a lot of time copy-pasting paths or generating these plots by hand, changing the colors of the lines and things like that. All of this obviously could be automated, or just is very well within sight if not done already.

Jackson: (26:40) On some level it sort of sounds like you're saying we will have laser beams and for the most part it will be people pointing the laser beams at something, but the laser beams are just getting better, stronger, and they last longer. It sounds like you're a little bit skeptical in the near term of the AIs knowing where to point the laser beams themselves.

Zhengdong: (27:00) I think it will happen. I just don't know when. Maybe compared to the optimistic AI researcher—the 2028 "country of geniuses" in a data center, takeoff and all— So I mean, on that, it will be like everyone has a country of geniuses in their pocket as well. But that's the difference. But on the laser beams, yes. I think we have these very high-powered laser beams. I think right now, since forever, we have had these laser beams that are just like human expert AI researchers. Ten years ago when it was an academic niche, we had a few very high-powered human laser beams that were copy-pasting the checkpoint paths themselves and doing this kind of search themselves. Because the models are general, because they really do replace tasks that are very useful to AI research and they're hugely complementary to humans, that's really all you need. We're expanding the number of barrels on the gun. As I mentioned, there's just so much to do. There are so many things to try that even with the incredible pace that we're building out compute, that talent is joining the field, and we're doubling the number of barrels on the gun, it still won't be enough. Now we just expand what we want to do. We look at all these fields of cognitive work and soon physical work, and we'll be like, "Okay, so now we need to optimize this for this particular task, for this particular job." You might think in the end, well, none of this is very general. You're just sort of brute-forcing your—

Jackson: (28:35) Yeah, it's the AlphaGo creativity thing.

Zhengdong: (28:37) Yeah, you're searching all possible—

Jackson: (28:40) Brute-forcing creativity.

Zhengdong: (28:41) Yeah. And at the end of that, you think like Move 37 is the obviously best move. Without having exhaustively done that search, it looks like genius to you. But if you had looked at all the millions of moves, then it's sort of obvious. You do this for all the spreadsheeting jobs, or eventually for invention.

Jackson: (29:05) Again, maybe I'm thinking about this wrong, but part of the way I'm almost thinking about this laser beam metaphor is that at some point we'll have big enough lasers that you can point the lasers at the end answer and you just say, "Run the search." Granted that's not actually one giant tractor beam. It's like a million different laser beams of so many different permutations. But if you scale it enough—

Zhengdong: (29:23) Yeah, no, I think it is a good analogy. One thing I'm surprised by—and I'm not sure how much other people are surprised, maybe AI researchers are too—is how easy language is. I actually do think a lot of other people will be surprised by this. Maybe in an early research agenda—I think OpenAI published it, I don't remember—they had all these bets early on in the company, including reinforcement learning, including robotics and things like that. It wasn't so clear that language would be the first thing solved; maybe it would be the last. The thing that we're doing right now, talking to each other in this space of language—it's surprising, or could be a little depressing, just how predictable all of this is. We could do so much of language—not the best poems, not the best novels—but so much of what we do with language is actually extremely, statistically predictable. There are these connections, or we can interpolate between the spaces. Maybe you could say the same about a lot of biology. Our human laser beams are just doing very little jaunts into this big unknown. There's so much we don't know in the universe. This very powerful statistical tool—the thing Demis Hassabis says in his Nobel lecture, his challenge is... But a classical algorithm will be able to solve all of these problems.

Jackson: (30:58) Yes. If you have a galaxy-sized calculator. Most complex problems are just a matter of time and search.

Zhengdong: (31:06) Yeah, and we could just be surprised by how simple it is relative to—

Jackson: (31:10) We're almost conflating that for this word we call "intelligence." Perhaps that would be a slightly strange but plausible view on what AI is, which is actually that there's no threshold of intelligence and it's just, again, big enough search and big enough data.

Zhengdong: (31:26) Yeah. I just think the whole AlphaGo, AlphaFold analogy is super, super useful. It kind of really quickly speed-ran through: there's pre-training, there's reinforcement learning. It's this very clean "what counts as winning a game" or "what counts as a value." You can just map it on these much fuzzier spaces that humans do. Then maybe at the end of the day, we find that all of the things that we humans do that we think are very complicated are actually like an extremely low-dimensional statistical thing in the entire universe. So we have these powerful things that could just find all the patterns really fast and interpolate all the spaces really fast.

Jackson: (32:09) What I'm sort of hearing you imply is that it's very plausible that most of the things we know a lot about, and our intelligence itself, might not actually be that complicated—and the universe might still actually be wildly, far more complicated than you can possibly imagine. It's possible AI speed-runs us and it still hardly knows anything.

Zhengdong: (32:30) Yeah. And I think Demis does a really good job of keeping us focused on these stakes. You know, this Silicon Valley techno-singularity where we're going to have Dyson spheres and extreme abundance. Even that is so small, or pales in comparison to the true nature of reality, which is that we have no idea what's going on in this whole universe. We have this chance of understanding a tiny bit, seeing some patterns, or touching the fabric of reality of the whole thing. So yeah, we could be more ambitious. Or that's even weirder. It's hard to—

(33:17) The Compute Theory of Everything and Feeling the Wave

Jackson: (33:16) It's humbling and inspiring at the same time. It's June 2026—I don't know when we'll put this out, but probably around now. I generally try not to have conversations that are of the moment, but I think it's worth grounding the time. In part, I want to talk about the main theme of your 2025 letter, which is... I would say it was probably the first thing I read where I, at least very strongly secondhand, felt the wave crash over me. Probably from using some of these tools lately. You basically lay out what you would call your compute theory of everything. You say, "However hard I try, I don't think my descriptions will even move you much. You need to pick your own test. It should be a problem you know well, one you've worked on for years, and one you're supposed to be an expert in. You need to predict the result year after year. Then you need to watch AI confound those predictions anyway." It's kind of a version of what we were just talking about, like actually, maybe language isn't that hard. I think we are in a different time than we were even in December when you wrote this. The US government has started to pay more attention, so on and so forth, and yet most people clearly still haven't felt the wave. What do you think? For every individual person, is it just going to be this kind of thing? Do you think there is a macro event that would convince people? Is it getting easier for you to convince people? Maybe a different way of asking the question, to use a metaphor from the piece, is: will there be a "March COVID" moment? Where in January and February it was available for people to see, but the world didn't wake up?

Zhengdong: (34:48) Yeah, this is a great question. This is something I keep thinking about all the time, too. I wanted to write this particular thing for the letter because I felt like it was so hard for me to feel the wave crash over me. As I mentioned, I maybe have longer timelines than the optimistic AI researcher. It actually took me five years of full-time being an AI researcher for me to become this "AGI pilled," so to speak. For this to happen, I had to see so much evidence before my eyes—evidence across time.

Jackson: (35:27) That first quote is the critical part of it. It's like, "I tried this now. I saw it was not that good. Two years passed. Oh, my gosh."

Zhengdong: (35:34) Evidence across time. Like repeated instances and tests that I set for myself. So it's not like you're telling me what this test is, or telling me it's really impressive. It's like, I know this problem. I know it would be impressive.

Jackson: (35:46) You have finger-feel in that problem.

Zhengdong: (35:46) Right. And I know you can't cheat on this in ways that I would know about. That's just such an expensive way to AGI pill someone.

Jackson: (36:00) To learn something.

Zhengdong: (36:01) You can't expect everyone to spend five years of their life—

Jackson: (36:06) —full-time, or shut down all the schools, in the COVID metaphor.

Zhengdong: (36:08) Yeah. There must be some accelerating way of this. When you're talking about the COVID analogy, I think one thing that will do it would be robots. That's maybe my upper bound: robots walking around.

Jackson: (36:25) It's not a perfect metaphor, but I remember the first time I got into a Waymo. I was videoing the Waymo, thinking, "This is the craziest thing that's ever happened to me."

Zhengdong: (36:31) Yeah.

Jackson: (36:31) I got in the Waymo, and 60 seconds later, I was on my phone and I had forgotten.

Zhengdong: (36:35) Yeah, that's a great point. So that also needs to be repeated instances where you're like, "Okay, so this week Waymo showed up, and then the next week there's these automated house-building robots that started building data centers in my backyard."

Jackson: (36:49) You have to be thrust into moving history, perhaps in a repeated sense. It can't just be one instance of moving history. It has to be—yeah, right.

Zhengdong: (36:58) I think that countries that have developed really fast in the last few decades—my parents' generation feels this very deep in their bones. They grew up in China, and when they were kids, there was rationing. They had meat once a month or something. Now they live in this big suburban house in the US, and it's really great.

Jackson: (37:20) Not to mention even so many parts of China.

Zhengdong: (37:21) Yeah.

Jackson: (37:21) Jasmine Sun had this line about the skyscrapers popping up like mushrooms.

Zhengdong: (37:27) Yeah. There's this picture of Shenzhen 20 years ago where it's just a field, and then 20 years later, there's skyscrapers. Matt Yglesias tweeted, "I bet this really ruined the character of the neighborhood," or something like that. Living through that over a long period of time, seeing so many changes like that, makes you feel it. So that's the upper bound of it.

Jackson: (37:50) It is a fundamental characteristic of humans. You don't see your kid's baby for three months and it seems like they've grown, like they've doubled in size, yet the parent doesn't notice it. For better or for worse, we are so good—we might hate change, but we're so good at normalizing change, which makes this jolting hard.

Zhengdong: (38:07) Yeah. I really have to credit people who are way more rational than me, or a community maybe that was or is rationalist-adjacent, or calls themselves it. They were early to the pandemic. They were early to AI. They deserve huge credit for that. I just can't get myself to do that; I had to see so much evidence. I really feel this difficulty. I feel very privileged to be surrounded by people who are very optimistic about AI, telling me all the time and trying to convince me of things. Finally, I was like, "Okay, I've got it." And now I'm—some people say—off the deep end.

Jackson: (38:54) Was that intellectual or more metaphysical, spiritual?

Zhengdong: (39:00) It definitely has to be a combination. There are these pieces of evidence I see for myself, but then you still need someone to handhold you through the rational look at the bigger picture, broadly across society. I tried to do this in the opening to my letter, and I keep trying new ways of doing this. If you just look at 10 years ago to today—10 years ago, the Transformer paper wasn't even out yet. 15 years ago was about when deep learning started working and AI could start recognizing cats and dogs. Just last year, the Pope named himself Leo because of artificial intelligence. He's like, "This is a revolution that deserves the same kind of shepherding that the first Industrial Revolution went through." That's why he's going to name himself Leo. Also, there are IMO gold medals, which are not even impressive now, six months into the year. Just that level of change—going from a small academic niche no one's heard about to the world spending close to a trillion dollars on investment and propping up the economy—is incredible. Now, in more recent, still unresolved political news, the federal government is realizing some of these models are really cybersecurity-dangerous. If you see this and you're not convinced, I'm not sure what would convince you. Looking forward, it's like, okay, maybe you've just started reading the news or just woken up from a coma. You're like, "I can see that AI is a big thing, but..."

Jackson: (40:39) I would argue if you had woken up from a coma, it would be easier.

Zhengdong: (40:41) Yeah, definitely.

Jackson: (40:43) This is the thing. I'm not referencing—it just came out that with GPT-5.6, the government, I think, is going to be hand-selecting who gets access to it.

Zhengdong: (40:55) Right? Every individual or something.

Jackson: (40:56) If you showed me that relative to two years ago, I would have been like, "Oh my gosh." But they just banned it or whatever.

Zhengdong: (41:03) Yeah, that's fair. I did try to say, "Imagine 10 years ago and then now." But say you see everything now and you're like, "Okay, I grant that AI is a big deal, maybe bigger than the microwave, but not a millennium-defining technology."

Jackson: (41:23) Even the internet was a big deal. I think we're starting to get to the point of the Industrial Revolution. I have an idea in my head that the Industrial Revolution was a big deal, but if I went back and lived through the change, it would be different. These things are hard in the abstract, which is part of why I asked the question: Is it purely an intellectual thing, or something else?

Zhengdong: (41:41) Yeah, so by the way on this, I think people should all be history majors and this helps a bit where, as part of the intellectual thing, you go back and you read stuff and a hundred years ago, people still felt like the end of the world was coming. You had chemical weapons in World War I, motor cars everywhere, and societal issues looming on the horizon that you don't know how to solve. America is only 250 years old. People have been just trying to make it work the whole time and abundance is a new thing. But on the rational aspect, if you look just a few years out in the future and you grant that AI becoming a bigger deal has been going on longer than you expected. Maybe one world is: you see ChatGPT, you see GPT-4, and you think, "Okay, this is obviously very impressive, obviously beat my expectations, but I'm not ready to sort of just go full straight line up and to the right on the graph." Because you're just more of an empiricist, you haven't seen enough data points. Then you just think, "Maybe it will keep improving for a few generations, but if it plateaued, it wouldn't be that weird." And so you're just waiting for more evidence. If you grant that now, you could say there's either five years or there's 10 years, or 15 years, or 70 years of continual dots on the plot that keep going up and to the right. Then the rational thing that I'm trying out to pill people is: how many more doublings do you think we need before this total physical transformation happens or bigger kinds of transformation happens? Even if you think the current level of investment is unsustainable—amount of capex invested or revenue or something like that—we'll just know the answer in the next five to 10 years. Right now we're in this sort of compute crunch because we need to build a lot of infrastructure. But even if the rate of growth slows, if it just continues, either this AI investment is going to exceed world GDP—which means it's going to have to crash—or we're going to have to grow world GDP because of the AI. Or we'll reach some crazy form of superintelligence and all this crazy stuff will happen, or there will be some—hopefully not—but some terrible world-historical geopolitical conflict or tragedy.

Jackson: (44:21) That's kind of the point you make at the beginning of the piece, though, around the Peter Thiel quote about Elon. For the people who haven't read it, your point is—or Thiel's point is—that Elon is talking about having a billion robots and all of these national debt problems. Thiel is critiquing that those two things aren't mutually exclusive along the lines of what you just said, granted, your point is that perhaps we did get the robots and the debt.

Zhengdong: (44:41) Yeah. But basically, the intellectual side of the equation looking forward is: I'm trying to tell you it's gotten so big now that we can rest easy. We will know the answer in the next five years.

Jackson: (44:54) It will either implode or we will enter a new epoch.

Zhengdong: (44:57) All of us are wondering what's going to happen. There are these people who are coping with, "AI is big, but it's going to stay this big," and things don't change very much or history has ended, so nothing ever happens. Things just generally stay the way things are. I'm like the Financial Times chart where one line is going up and one line is going down. We'll just know the answer. It's gotten big enough where this will happen very soon, and so it's very exciting. I don't think it takes that much of a leap, once you accept that AI is a real thing today and that it has changed a lot in the last few years. We'll just know what happens. Or it's sort of like knowing a pandemic-level event is on the horizon. We just don't know if it will be very good or bad.

Jackson: (45:49) I don't think that's going to inspire much confidence in people, that last line.

Zhengdong: (45:53) Right. But maybe this exercise is just to get people convinced about the moving history instead.

Jackson: (46:01) Well, so there's one line in the piece that I think is really poignant. You say at one point that the curve is exponential, so if it happens, it doesn't matter when. And then you go on to say, "What does it matter if it's two or 20 years away? The only change that's mattered to my AI timelines is that I used to think it wasn't going to happen in my lifetime and now I think it is." Can you talk about how that view has most changed your research and the way you think? Part of the implication of that is that we're not going to have the debt implosion thing or the economic implosion thing in the next five years. It's going to happen. We're moving up the exponential—if we are moving up the exponential at all—like we're going to clear the jump, maybe. But why is that framing of it? Obviously, on one sense, it's just fundamental in that you're going to get to live to experience it. But I think there's a subtler frame on it, which is that if we're on the exponential, it's not slowing down.

Zhengdong: (47:02) Yeah, maybe I would think about it—and this is extremely relevant for everyone's personal life, whether or not you're in tech, whether or not you're in AI.

Jackson: (47:12) Granted, the two or 20 matters. Two or 20 years is a big difference.

Zhengdong: (47:15) It definitely matters. But overwhelmingly, most important of all is: are you on the exponential or not? Do you recognize that? If you can just do something to tie your ship to the exponential versus just knowing it, you're probably going to make broader, better decisions.

Jackson: (47:40) Sorry to interrupt, but I think that's quite an interesting question. One other meta reason why people aren't able to feel the wave crash over them is—even if I do look at the math, which is just look at the scaling laws—the root point you make over this whole piece is that we're on this long journey, arguably back to Moore's law or even further. One reason perhaps that we don't feel it is: what the heck am I supposed to do with the fact that... what does hitching your wagon to that even mean? Does it mean go work in AI?

Zhengdong: (48:16) Yeah, I think any of these specific things are probably going to be wrong. I was talking to a friend about Knightian uncertainty—but I like the Donald Rumsfeld version better. He goes up to the podium and he's like, "There are known knowns, there are known unknowns, and there are unknown unknowns." In the unknown-unknown situation, you really don't know what the optimal strategy is based on just some deep preference you have. You might be extra greedy or extra risk-averse, but both of those seem just as good to me. I think that within AI worlds where we're convinced that this is going to be a big deal, we spend a lot of time talking about whether it's going to be two or 20 years. Like you said, it's super important; it will totally change how society is able to adapt well to it or not. But then I think that for someone else, the default, "haven't really thought about it" reaction is just, "Okay, I just really hope this doesn't happen." But if you look at all the beliefs you have or which conclusion you draw—if you just look at the evidence and apply your own priors to how you would weigh the evidence—you might come to the uncomfortable conclusion that this will happen in the next 20 years, or the next 50 years, just sometime in your lifetime. It definitely matters when, but I think it does give a lot of clarity to things. It just accelerates you asking the question of what do you value?

Jackson: (49:46) Yes.

Zhengdong: (49:46) And so now you could say, okay, if you're a young person, "I need to get a job, I need to save enough money, I need to figure out a bunch of things in my life." Right. And then when you're young, you think you're going to live forever and you never really think about the longer-term things. But I think the fact that this phenomenon exists—even if it hasn't really physically affected your life at all, you're totally far away from the AI world or something—this is sort of like telling you there's going to be a pandemic level of change in your lifetime. It's a totally valid question: what am I supposed to do about that? But maybe one thing is just that it becomes a lot easier to stop caring about the petty things because you're like, "Yeah, I've just received this prognosis that things are just going to be very different in five years." You're going to look back and you're going to be like, "I wish I didn't care so much about what other people thought of me," or "I wish I spent more time with my friends," or something like that.

Jackson: (50:53) It does feel—on some—the more I think about this, it does feel like if I told you you were going to die in a year, or anyone was going to die in a year, especially if they were under the age of 80.

Zhengdong: (51:01) Yeah.

Jackson: (51:01) I suspect they would really change their behavior. If I told you you're going to die in three weeks, sure. But if I told you you're going to die in seven years, yes, intellectually I think I would change. Versus 40.

Zhengdong: (51:12) Yeah.

Jackson: (51:12) But we're really bad.

Zhengdong: (51:16) Yeah.

Jackson: (51:16) Letting abstract, remotely far-away things... letting the wave crash over us. Perhaps a question would be—

Zhengdong: (51:25) And unfortunately, I think this is a really good analogy. I'm just desperately trying to find the positive versions of these analogies. There's like, you receive a terminal diagnosis, or the whole of society goes through a pandemic. But I really do believe that AI is going to go well, or I think it's way more likely to go well than not. And so while these analogies work really well because they really hit on some sense of urgency or make you feel the stakes, the "imagine you will win the lottery next year" or something like that—

Jackson: (52:09) By the way, if I knew I was going to win the lottery in three weeks, my behavior would be very different.

Zhengdong: (52:15) And so the key to finding such an analogy—you as an individual winning the lottery, or society as a whole winning a society-wide lottery—is how to make that more compelling or how to communicate that. I think the whole field is also struggling with this, where the bleak things just hit home more in some way, unfortunately. But whatever you would do there, it's like, "Yeah, imagine you're going to be fabulously wealthy in a few years." Like you said, you would be acting very differently, or you would receive some kind of new clarity about what you care about or what you value. Or you would just know, like, "I need to really quickly start thinking about what I value, what makes me human," things like that. I think the labs have achieved this. That's why they're starting all these organizations thinking about all of this—the transition and also after the technology is more stable.

(53:17) The Myths AI Needs, Competition vs. Power Concentration, and Whether Progress Is Inevitable

Jackson: (53:18) One major implication of this is that the people who do know this—who have this secret, or not-so-secret, people working in AI—need to be better at talking about it. It's clear that that's something you think a lot about, both in your personal writing as well as what you urge people on. There are two quotes that I picked out on this that I liked. The first: "Some people are quick to disavow themselves from doomers or accelerationists. But what else is on offer? Not much. If research engineers continue to see myth-making as a chore, second-class, a lesser use of their time to 'real technical work,' they will keep working in a world under myth they keep complaining is inferior." And then, slightly more fun, you quote Tolkien: "Fantasy is a natural human activity. It certainly does not destroy or even insult reason, and it does not either blunt the appetite for, nor obscure the perception of, scientific verity. On the contrary, the keener and clearer is the reason, the better fantasy it will make." What makes for good myth, maybe to start?

Zhengdong: (54:32) Yeah, I think there is a really good myth specifically for people who are likely to become AI researchers.

Jackson: (54:40) Right.

Zhengdong: (54:42) All these labs have lots of very talented people working for them, and they could go into finance and make a lot more money or something like that. But it's partly because of the other mission-driven people, and the mission itself. Whether it's touching the fabric of reality, understanding the true nature of reality, or impacting a lot of people's lives—like curing cancer and eliminating poverty, hopefully eliminating inequality—that's a myth, in a good way. It's a mission that really drives people.

Jackson: (55:24) It's such a strong myth, perhaps in part, that it could be why these people aren't thinking about that much myth-making otherwise. It's just so intrinsic and obvious to us. It's catnip to a certain kind of person that they're just like, "How could you care about anything else? This is the most important thing."

Zhengdong: (55:41) Yeah, exactly. And also the myth of, "Oh, you are on the Manhattan Project, you are Oppenheimer, you are contributing to this world-historical event individually." That also works. So I think in that way, the myth-making is going quite well. It's just that this is definitely too simplistic of a view, that there's not enough understanding of the diversity of the fantasies that would appeal to more people. Maybe it's something like that. Maybe it's that change is just really uncomfortable because uncertainty is really uncomfortable, and generally, people don't really like uncertainty. So maybe finding the perfect myth, or finding the Steve Jobs for AI, is just a really hard problem. That could be part of it. I think it's definitely worth working on and thinking about, though. Right now, the best myths are directed at the people most likely to receive them. You receive it, and you might join AI. As AI becomes bigger, it becomes more inclusive, more pluralist, more normal, so to say.

Jackson: (57:00) Two questions. Number one would be: in that quote, you reference doomers and accelerationists and "what else is on offer." I don't think you're a doomer; I feel pretty confident on that. And maybe you're an accelerationist by some definition—certainly to a normal person—but I don't get the sense that you're a pure accelerationist. I think you're working on this in part because you think it's the most important thing. Do you have a sense of other narratives that are on offer?

Zhengdong: (57:28) Yeah, so this quote was from a couple of years ago. I think that was the year where maybe e/acc was coined or something like that. In my mind, that was the year newspapers started to use the word "doomer." I chose those two as the most prominent myths of the time, but they're still definitely ingrained in the water. I think there are new, more complicated myths being developed. I would agree with you. If I had to choose between these two ends, I would say I'm more of an accelerationist because I think it will go well. I think you should really focus on the "solving cancer 10 years earlier" kind of thing being really good, rather than trying to avoid some bad thing that you think will happen earlier. These myths have also gotten more inclusive, so there are different strains of accelerationism now.

Jackson: (58:39) The second part of my question was going to be: You've also spent time, as I understand it, with a number of people who are closer to the policy side of the world, certainly in the UK and maybe also in the US. Do you have a sense of what types of myths or stories or even just simple messaging have been or will be most resonant or clarifying for those people? It feels more relevant than ever, at least in the US. Some of it's about persuasion, but some of it's just about making people feel the wave. You make a point somewhere—granted, I don't think this was that recent, but it wasn't that long ago either—that there's a few people in the Trump administration, a few people in Saudi Arabia, and maybe a couple other senior politicians in the world who have felt the wave.

Zhengdong: (59:31) Yeah, not my point, but I was quoting someone.

Jackson: (59:34) Okay, I'm sorry.

Zhengdong: (59:34) Yeah, so I think there are two maybe reassuring things. One is that this is not really change. In the thing I wrote where I cited this person, the argument is that you should frame it as: we are keeping a promise that we previously made. The technology that we use to keep it is different, but broadly, we have a promise that is the social contract, and we will uphold it by distributing the benefits of this technology to everyone. We have to make sure no one's left behind, but fundamentally, the important stuff has not changed. You're living your life, you get a new house, but everyone is still going to get a house. The change will not be so disruptive or so uncertain that you need to really worry deeply. The second thing that could be reassuring is that volatility is good. We've always had changes, and change has always been good. It's like if you grow up in a country with 10% GDP growth over decades and you just watch where you live turn from a field into skyscrapers, and all of a sudden you have these really fast bullet trains. Yes, there's a lot of change, but broadly, overall it's good for you. I think it works for America, too. America is only 250 years old. You look at some guy who's alive today and just three generations back was the ninth President of the United States. It's just not that much time. For most of American history, so much happened. For most of American history, America was not the global superpower. So that's a lot of change. Telling this myth where you are empowered to benefit from this change is important. We would hate it if we had an extremely classist society where nothing ever changed and you're just kind of stuck in whichever social standing you have. But you're part of a country of very dynamic people who are starting businesses and doing all this cool stuff, and you're making the change yourself. You will benefit from this change as has happened throughout history. In fact, it's part of your national identity that as a person of this country, you're really good at adapting to change, and change has always been really good for you.

Jackson: (1:02:21) There is a structural—perhaps this will change, but currently it certainly seems that there is something structural about AI and maybe just modern kind of techno-capitalism that is creating a smaller and smaller group of really powerful actors. Maybe a different cut on this would be that's one justifiable concern. There's a different take you have somewhere—and I think you're kind of referencing somebody else's idea—but it's the person who's talking about the French Revolution versus the replacement of the English monarch. It's almost like it's definitely a little Machiavellian or something, but it is an interesting way to think for whoever's trying to change the world. You best keep some of these promises. You say—I think this is you: "The AI industry has all kinds of French revolutionary tendencies in our own self-conception. We're bold, inevitable, and on the right side of history. At our worst, we're ignoring our inheritance to remake the world from abstract rational principles, dismissive of accumulated experience and impatient that no one else is keeping up. I think we're meeting a resistance to that impulse that is earned."

Zhengdong: (1:03:26) I could go back to my classic cop-out answer of both have to be true at the same time. I really believe in competition being good and if you want to prevent really serious power concentration, there just needs to be a lot of competition. So, a lot of unpopular decisions that labs have made—people complain about them. Maybe they made these various decisions because they feel like they're in a position of strength and they're like, "We think this would be best to do with the models, and if we do this, the market can't do very much about it, so we're just going to do it." But then because there's competition, maybe that gets walked back in the end. Maybe labs would do other things that they're not doing, such as, "We're just going to develop this RSI AGI thing by ourselves and just keep all the innovations to ourselves." But because there's a market, that's not going to happen, or there's a lot of competition between different models. So one company's idea of what is moral and good and how much to defer to the user versus how much input the model should have to the user—the model telling you something is wrong or something. All of this, I think competition is generally good. But then, of course, that means maybe this pulls the future closer to us a bit faster and you're like, "Wait, but that's bad." But both kind of need to be true and happen at the same time. So, I appreciate you muddling through this with me.

Jackson: (1:05:05) I mean part of the meta thing that I think I'm feeling as I've been thinking about this and through this conversation is just all further case to be made that the more we can make a plurality of different types of people—Americans, politicians, whatever—to feel the wave and thus really deeply engage with this. There's a tension there, of course, because one response to that might just be like we need to stop the AI. But the best possible future to me seems roughly like as many influential and regular people as possible care about this stuff. They have the optimistic understanding of what the technology is, but they also aren't just putting it off as this pie-in-the-sky thing to say like we have two to 20 years to solve some of these problems and it's going to take the whole gang to really...

Zhengdong: (1:05:55) Yeah, no, but we should focus on—it will take the whole gang. All the labs, I think, are extremely sincere and well-intentioned. What we know is that it's going to be nuts. And we're not professional economists. Well, some of us are, but not as many as the whole field of economics or the whole field of philosophy or the whole field of whatever it would take to go well for all these different areas of human inquiry. And we're just kind of creating these organizations to think about these questions because they are important questions. But yeah, some kind of main goal is to get more people into this project and it will take the whole gang. If you don't like that big tech is deciding all these things, then just get involved. And you can't consistently hold that this is a scam and also, "Oh, please stop disrupting my field."

Jackson: (1:07:06) Maybe just briefly—I know we briefly touched on it—but why aren't you a doomer?

Zhengdong: (1:07:11) So I think I am still more of an empiricist and therefore there are a lot of risks I do worry about. And maybe this requires a definition of doomer. But yeah, a lot of the risks are very much where humans are involved in them. Malicious humans using the AI for bad purposes, I think, generally characterizes the risks I'm worried about rather than the...

Jackson: (1:07:40) Paper clipping, whatever, we lose control.

Zhengdong: (1:07:42) And some other general assumptions I'm making are like there will be competition. There is just going to be a lot of degrees of freedom for a lot of different things that you can't predict ahead of time to happen, or avenues of these threat vectors. And so given that we live in this very decentralized world, that we've made choices in society to support the benefits of the decentralized world, then you really need to lean into it. And so if it is a spectrum of accelerating versus slowing down, then I think most people are good, most people are not malicious. And of course there's offense-defense balances. But the kind of solution that I imagine a vague definition of doomer would support, I feel like would be counterproductive. And I'm not at all saying the risks aren't real. I'm saying I put a much greater weight on these more human-being-involved kind of risks, and that the best solution to them—like I wrote in the piece responding to Dara's piece—is very decentralized, with a lot of good people in a very decentralized way thinking about this.

Jackson: (1:09:06) Yeah, there's a cut on this that's sort of like the cost of progress and the cost of liberty is some risk. And we have taken that trade for a long time and it has paid off really well.

Zhengdong: (1:09:21) And I would add that the non-liberty is a bigger risk.

Jackson: (1:09:26) One last thing on this, a quote I liked—and maybe just another opportunity for the wave to crash over people a little bit on compute and scaling. You say, "The biggest mistake people make when they make the case for AI is that they say it's different this time. It's not different this time because it's always been different. There hasn't been any constant normal trend ever. And all we've done is be optimistic that we'll muddle through. Nothing is truly inevitable. Certainly not progress. And progress, too, might seem to stop tomorrow. All things considered, though, it would be stranger if it did than if it didn't." Perhaps you even make this point, but I couldn't help reading this thinking about how not only does this apply to scaling laws, or possibly even computing and Moore's Law, but maybe all of human progress—maybe even the trend of complexity in this little corner of the universe that we're in. You also noted your frustrations with the Foundation series. Speaking of determinism, how do you think about the sense of inevitability or determinism on one hand, and also will or agency on the other?

Zhengdong: (1:10:40) Both are true at the same time. But more seriously, when you look at the trend of 2% GDP growth since we "invented invention," if you zoom in really closely, it's all these people trying very hard and doing very different, weird things every time. Only when you zoom out does it look like what we call a law. If you believe that AI is general or is extremely complementary to humans in some way, it's like these human laser beams that we're focusing on processing problems. As a general rule, we know that if we focus this human laser beam on a problem for some amount of time, we're going to get something out of it. Of course, it's not guaranteed. You're probably just rolling dice, but if you put enough human laser beams on something, you get a result.

Jackson: (1:11:36) We even see this with things like the space race in the '60s or COVID vaccines. There is something about human drive.

Zhengdong: (1:11:44) Individually, too—the will to power. As a researcher or as a creative, you're like, "Damn, am I really going to be able to think of a next piece?" But if you just block aside 10 hours, you're going to get something out of those 10 hours. It's going to be a bit more or a bit less than you expected. In that way, everything is different. Everything could definitely be zero. So that aspect is extremely contingent, not determined. Then when you zoom out, it looks more determined. It's both. The thing about the Foundation series is I just think there's no character development because there are these hundreds of years of time jumps. You only get a character for a chapter. I think the idea is very cool and very important, and a science fiction book should really lean into its one idea. It was hugely impactful and maybe captured something about the time it was written, where people imagined, "Oh, we have Newton's laws. We're discovering all sorts of laws. Maybe we'll discover laws to everything." That's a very old idea.

Jackson: (1:13:04) Amazing point.

Zhengdong: (1:13:05) To have a work of fiction that really explores that is great. I'm not anti-Foundation just because it has that idea. I think it also relies on the reader to not go off the deep end. You read that and you think, "Okay, now I'm going to discover the true law of history." But it's just a data point that you take in your own balanced "everything is true" sort of idea. It's just another character in the pluralist universe.

(1:13:42) "Who Could Possibly Compete?": Post-AGI Meaning, Work, and Questing

Jackson: (1:13:42) I want to talk about personal implications of where we might be going in the era to come. First of all, on a similar note to what we just spoke about, I wanted to read two excerpts. First, from your 2025 letter: "Later you'll think, who could possibly compete? How could your cleverness be worth anything more than a hill of beans against an artifact that cost millions and concentrates within it the cleverness of billions of humans? How arrogant to think yourself clever enough to outpace a factor of a thousand, then another thousand, then another thousand. This is what they mean when they say general-purpose technology." That is a very good articulation of the "super laser beam." Meanwhile, in your 2022 letter—the first one available to me—there's a slightly different tone: "Some people think that there are few, if any, scientific breakthroughs remaining. They think research progress is hard to measure. Ideas are getting harder to find. Maybe all the good ones have already been had. Maybe some 'X is all you need' is all you need. I wouldn't be so sure. Marvin Minsky, luminary of our field, predicted, 'Within a generation, the problem of creating artificial intelligence will substantially be solved.' That was in the late '60s. He joins a distinguished class—Lord Kelvin in 1897: 'There is nothing new to be discovered in physics now. All that remains is more and more precise measurement.' Cicero reported that Aristotle thought he had just about completed philosophy and that it would surely be completed a short time after his death. Don't be them. Here, actually take the long view." We're coming back to it over and over again, but there is a paradox there. One is empowering to the human spirit and the other is deeply humbling, if not disappointing or even sad. Do you still believe in the Zhengdong of 2022?

Zhengdong: (1:15:45) Yeah, I can see the tension there. But I think it is consistent in the reason why you are trying to be clever when you do the research and when you're trying to compete against the models. If you're just doing it for some instrumental purpose, like we do in a lot of the things we do in daily life and errands and for our job—where we need to get this thing done and here are the things that we do to get there—that's one thing. In a job, you might think, "I'm just having a blast every day doing this thing," and that can all be consistent and great. If you want the end goal itself, then you can use a tool to accelerate the middle part that maybe you don't care so much about and reach the end goal. But let's say in terms of software engineering, you also like the process itself; you like the puzzle-solving aspect of the engineering. Maybe it is a bit disheartening that you don't find it as useful to other humans because there's this tool that could do it. But you can still explore this kind of puzzle-solving aspect yourself in the same way that athletes do or game players do. Autotelic games being a very self-imposed limitations kind of thing—"Can I solve these problems within these limitations?"—or as an artist. Now you can do software engineering as an artist. The other aspect that I think is new, that's more consistent with the earlier, more human-empowering thing, is that there's just so much stuff to do. Part of my earlier answer about getting AGI not being what it's cracked up to be is that if you just think of the amount of things we will demand, or the amount of things that we will explore—just how big the universe is and how weird it is—there will always be something to do. Scarcity in the future—the definition of it: Will the cost be zero? Will we really have infinite of it? We won't reach that. We'll just do everything faster. We'll do more things, get even more niches, write even more fiction, and create even more universes for ourselves. The second way that there will always be stuff to do is just something being very personal. Everything about you—maybe you like to do puzzles or you like to create this sort of art—just the fact that you are doing it yourself is also very new. So whether you do it with AI or you do it by yourself, people are fans of your work just because it is you. Right now, friends will read your stuff because they want to know what you think. They want to know your favorite flavor of ice cream, not the LLM's objective, optimal best flavor of ice cream. Many people have written about this as well. So the personal stuff is always going to be your domain. Scott Alexander has written a very good post that makes you feel this particular wave crashing over you. Imagine in the future if there are people in other galaxies and they read all the history of what the creation of powerful AI was like. They're like, "Every single character that was even marginally related is like a celebrity." "Oh, this was their favorite form of ice cream. This was a blog post they wrote." It reminds me of in the Bible, the first European convert to Christianity—her name is Lydia—is just remembered forever because she's living her life and is part of this world-historical story. So the personal is new. There's just infinite demand for everything. Yeah, there are only some particular things that, because they're accelerated by tools, you would do differently. But I just see everything as an expansion of options available.

Jackson: (1:19:52) I agree with much of that. And yet part of the implication in that 2022 quote, I think, is about doing things of substance. Granted, I think relational life is of substance, but in the abstract sense or in the grand sense, it would be like discovery research—truly getting closer to knowing. In some sense there is a humbling, in a positive way, the idea that actually we know 0.00001% of what there is to know and we can know so much more. But that isn't really something you mentioned. Then, I guess at a more local level, most people aren't going to discover scientific theorems. I do think there's just a sense that for many people, it's hard to have meaning without some kind of meaningful work.

Zhengdong: (1:20:40) Yeah.

Jackson: (1:20:40) So I'm curious how you would take those two on top of what you said recently.

Zhengdong: (1:20:44) Rebecca Lowe wrote a blog post—she's a philosopher—about some broader definition of work. She discusses this idea of replacing jobs, but what even is a job? I think she and I would agree with you that everyone needs to have some kind of purpose. Maybe it's just relationships, but maybe it's not just relationships. You just need to feel like you are doing something that you find meaningful or you are contributing to something larger than yourself—a community or a bigger project. That can be very broadly defined. I'm curious if you think that people's idea of that can change or not. Maybe with the invention of some tools for agriculture or for making clothes, previously our culture would place much higher weight on being able to hunt or being able to farm. There's something real about waking up early in the morning and doing this work that's "real work." Maybe even today we idealize this idea of "real work" instead of spreadsheets.

Jackson: (1:21:49) I was literally at dinner last night at one of those open kitchens, and I was like, "Wow, it'd be kind of cool to work in a kitchen for a little while." You remember?

Zhengdong: (1:21:58) They were building an apartment building next to the office a few years ago, and a colleague and I would joke: "We're just coding, but if you want to see any real engineering happen, just look out there."

Jackson: (1:22:14) I think I agree. And yet, if you take one single thread throughout all of human history—to my knowledge, certainly all of known history—there has been a frontier to explore. Perhaps, if anything, the last 30 or 40 years has been an anomaly where the frontier was the internet. Now obviously it's extended to space. Some people will be totally fine in relational contexts. Some people will be totally fine making good pants, making music, whatever. I make a podcast. But there is a sense that there is some deep human thing—it's hard for me to imagine it going away, at least as a species—of our desire to have quests and grand adventures and to be of consequence. Perhaps that last bit is the part that was made up all along, or where we were deceiving ourselves all along. And maybe this kind of goes hand in hand with a broader question: on some time horizon, do we get superseded in the local context of what intelligence means over here?

Zhengdong: (1:23:22) Maybe. Correct me if I'm wrong, but I would say an assumption behind your question is that there's this space of questing—that your quest could do this, that there's all this unknown, and because of AI, AI is just taking up a lot of this space of questing. It's like 90% of all the questing could be done.

Jackson: (1:23:45) That could be theoretically 100% of the globally relevant questing. Maybe we'll still have locally relevant questing, right?

Zhengdong: (1:23:51) Yeah.

Jackson: (1:23:53) Or maybe we never had globally relevant questing.

Zhengdong: (1:23:58) Could you talk about this assumption a bit? Because my assumption would be the idea of 100% of the globally relevant questing would itself need some justification. It's your burden of proof to show, because as long as everything isn't instantly completed at zero cost or something... Let's say your quest is, "Oh, I want to go to that galaxy over there and I want to explore everything, and I'm taking all these robots and starships with me." When you play a video game, everything is done for you, right? You still make these vague decisions, and your choice of, "This is who I want to be; this is my identity," is still your choice. There are still choices that literally only you could make.

Jackson: (1:24:53) Is that true for science? Maybe. Let's put it a different way. Well, a few thoughts. Number one, the quote I read which is like your cleverness—later you'll think, "Who could possibly compete? How could your cleverness be worth anything more than a hill of beans against an artifact that cost millions in concept?" Thousands and thousands... You give a different example: we talked about AlphaFold or AlphaGo. Is it actually creative? Well, it doesn't really matter. A third thing I would say is, you are someone, I feel quite confident, who could just go enjoy themselves all the time, right? And eat food and visit art—and you do a lot of that, fortunately. And yet you are also like, "No, I have to work on this." And maybe part of it is that there are only a few more years, but we want to matter. We want to matter.

Zhengdong: (1:25:42) Right. Or if you ask me personally, maybe it is true that there is some kind of objective definition of what morality is, or just an objective answer to everything. When you discover the true nature of reality, there's this theory of everything, one equation or something like that. Maybe that is the case. To take what you're saying really seriously, I think that if we were to find such a thing and we were to all be very deeply convinced by it, that would be concerning. That would open us up to exactly the kind of worry that you bring—that everything is determined, there is nothing else that really mattered, and we've done 100% of the things that really mattered. You could think of it like that. You could also think of it as like the completion of the game—like Minecraft, you've killed the Ender Dragon, credits start scrolling, and there's this nice poem written there. That would be like the completion of the game of the universe. In Demis Hassabis's biography with Sebastian Mallaby, he says once he knows that, then he will shuffle off his mortal coil, or something like that. Maybe that's one way to salvage it. I agree that in that situation, it would open this up to your concerns. But I think we are just so far away from that. Even if our lifespans end up being much longer, I think a lot of it will just look like, "Okay, I've decided to take my starships and go to that galaxy," and I'm going to make a bunch of choices along the way that I find quite meaningful.

Jackson: (1:27:33) I think that's right. I think on some level there is a fear and a plausible reality that we stop being the most significant actors in the discovery of knowledge and the exploration of the universe.

Zhengdong: (1:27:46) I think this is all just relative to what previous culture was or what we grew up in, and we maybe find it hard to change our own preferences. If we grew up finding a lot of meaning in fixing up our house or tending to our livestock, then every single generation thinks, "Ah, the kids these days... everything's getting worse and changing." Now, another way in which it's not different this time—because it's always been different—is that these kinds of changes are just happening faster and faster. Where, yes, the short term can be a lifetime, but also it was great when the short term was longer than a lifetime or longer than the characteristic length of a career. But now, if you need to transition in a space that's shorter than the average human career, that's a step change in new problems. If you only had to do it once because it was in the middle of your career, fine. In that way it's different. Maybe now you're like, "Okay, this uncertainty is really worrying. I'm going to have to change my life to adapt to this uncertainty." And then three years pass and you're like, "I did it. I found new sources of meaning; I found new things that matter just as much as what I previously did." Then bam, the AI can do that now and you have to do it again. So maybe another general personal thing is being a lot more okay with change in a meta way.

(1:29:28) Are You Having Fun? Vacations, Mattering, and How to Live

Jackson: (1:29:28) I have a quote from you: "In my time as a research engineer so far, I have enjoyed many—too many to count—meditations on research taste. My only contribution to the literature is this: Are you having fun?" My question to you is: are you having fun?

Zhengdong: (1:29:41) I am having a blast. Are you?

Jackson: (1:29:44) I am. I am. I think part of this future we're talking about is how much fun you can have by surfing the waves of change. There are times where I feel afraid, and there are times where I feel confused, or like the risk of change is that if you ever respond to that with a desire for inaction, I think you can spiral in a negative way. In some weird sense, it relates a little bit to the classic Dan-Nabeel "do more and you'll get more energy" thing. So, I think there are ways to respond to all this and be paralyzed, and there are ways to respond—to be like, as you said, there's so many galaxies to explore.

Zhengdong: (1:30:28) Yeah. The galaxies, I think, are still quite far. But the having fun is like there's always this type 2 fun. People say you go on a hiking trip or there's some experience where you have a lot of uncertainty, and then you look back and there's some nostalgia, or you were glad to have gone through the experience. Of course, the experience has to go well. You can't ignore that. If you're hugely uncertain in the future because a lot of the things you relied on—that your life depends on—are now in flight, that is really bad. But I think it is a privilege to live through interesting times.

Jackson: (1:31:09) Are you having type 2 fun or type 1 fun? I was recently asked this.

Zhengdong: (1:31:15) I think it is a lot of type 2 fun with very short feedback where you reach the end, where you look back.

Jackson: (1:31:24) You're climbing lots of hills.

Zhengdong: (1:31:24) Yeah. Wow. Even now, I think back to a few years ago when ChatGPT came out. Imagine the first time I tried ChatGPT, and before that, the first time I tried some kind of language model that was not ChatGPT, or—imagining, mentioning before that—trying image models like DALL-E. Just how different everything was then. It's like, "Wow, did I really live through this and see it happen and talk to people at the time it was happening?" That all feels so far away, even though it wasn't that long ago.

Jackson: (1:32:05) What do you mean by that? And why do you identify as a consumer?

Zhengdong: (1:32:09) Yeah, so I was thinking of this when you were asking me if I'm having fun. By "consumer," on one hand I mean the character you read about in your economics textbook—you know, there's producers and there's consumers. I'm just the consumer; I'm just consuming this stuff. And then there's a broader definition. I really mean everything—not just a nice restaurant or a good movie, but also rarer forms of consumption. I feel extremely privileged to know some of the much better AI researchers than me—big characters in the field, the people who will map onto Oppenheimer and Rutherford and all these aliens in the—

Jackson: (1:32:58) Future galaxy will be reading about.

Zhengdong: (1:32:59) Yeah. Or later you read history textbooks and these people are characters. To know how they feel about certain things, to have been able to get their thoughts on something—that's just an extremely rare form of consumption that I feel very, very privileged to be able to partake in.

Jackson: (1:33:24) Is research a type of consumption?

Zhengdong: (1:33:27) No, I think it's production. Directly, you can get satisfaction and fun in discovering a fact and keeping it to yourself, and that can be a form of consumption. But I think producing is when you've discovered this fact and it's very cheap to share it, so you should share it. One thing Tyler Cowen has said before—I think it was him and I've just stolen it—he's like, "I just produce so that they let me consume." And we're both selfish in this particular way.

Jackson: (1:34:05) I was going to say, so at least you're producing a little bit. Most people's problem is that they would lean over too much into consumption. Perhaps in your milieu, and certainly in San Francisco, it's the inverse—people only work. How do you strike that balance?

Zhengdong: (1:34:17) Yeah, maybe there is not a perfect balance because I definitely have this Protestant work ethic sort of thing. I feel like I need to deserve being alive every day, and that I should probably just be producing 100% of the time just to deserve being alive and deserve being as lucky as I am. And then part of that is selfish in the way of, you know, they won't let me talk to these really cool AI researchers if I don't produce anything.

Jackson: (1:34:48) Consumption is unlocked.

Zhengdong: (1:34:49) Yeah, it's like you've interviewed President Bush's personal assistant. That's extremely rarefied consumption. I so look up to that. It must be so cool. You get to try out, on Air Force One, what kind of special meal they have that day. You get the little napkin with the presidential seal on it. But they don't let anybody do that. You've got to be producing something—either writing as a White House correspondent, or guarding the president, or being the president. That's how you get to consume that. And so, I guess that's kind of what I do here. I'm producing just a little bit, so I get to be in the room.

Jackson: (1:35:34) Yeah. And it's super fun. Maybe this is how the market works. You want to consume this thing? Well, then you have to produce the thing too. The price of consuming this is producing this thing. I also think one should not be totally selfish. Production is good; otherwise, we would not have all the wonderful things in the world.

Zhengdong: (1:36:07) The automated laser beams are going to be doing it all for us.

Jackson: (1:36:10) And for most of the time, people wrote all that code by hand. That was a lot of production. I bet even if they enjoyed it, they might have been writing slightly different code that wasn't all for the one goal and purpose of being able to consume better. What do you say to people, probably people who are more likely to be your peers, who feel that there's no way they could possibly take time off or take a vacation in these pressing times? To take the other side of that question.

Zhengdong: (1:36:47) Yeah, a bunch of things. Because we're on an exponential, now is better than ever. Now is, in fact, the last time. Second, it might be for all of the normal reasons why it's good. Humans still operate on a certain timescale, and if we are packed end-to-end every day, we really aren't doing any thinking. We're probably just inundated by all the noise and information, and it will literally be worse for you. The whole thing I could say about mental health and that being the most important thing—if you're thinking about this, you should probably just take a break. It's just a win across the board for everything. Practically, you could vacation for a year and come back, and your $20 a month subscription to AI will just be 10 times more effective.

Jackson: (1:37:36) But what about the people who are working at these places? I mean, I've definitely talked to a number of friends in AI who are like, "I literally can't imagine taking a day off."

Zhengdong: (1:37:46) If you're having fun, that's great. I'm just saying I don't think the cost would really be that much. And yes, I'm sure everybody is contributing their own subjectivity and the whole project is very slightly different, but you don't matter that much. But you do in the way that if you take a break, the economy without AI will still grow at 2%. Right. And both are true: You do matter and you don't matter.

Jackson: (1:38:18) I think that's right. Quote from you—I think it's possible it's the 2025 letter or one of the more recent essays. Jonathan Malesic in May reminded us that AI cannot teach us how we want to live. He writes of the humanities: "I will sacrifice some length of my days to add depth to another person's experience of the rest of theirs. Many did this for me. The work is slow. Its results often go unseen for years, but it is no gimmick." I think we've hit this point probably plenty, but do you have any advice for people on how to live?

Zhengdong: (1:38:50) Yeah, I think that's a beautiful quote by him. I quoted it at length, and it's a great piece. I do think when he wrote that post, he maybe laments AI a lot more than I do, and so I wanted to include it to contrast some of the more optimistic about AI stuff. But when I read that, I think of it as the choice of making something matter. In the thought experiment you gave me earlier, where there was this writing thing, maybe you were doing it so that you could write things that other people would enjoy reading. And now there's this machine that can write things that other people enjoy reading more than what you were doing. And so that would feel like a loss of something that matters to you. But at the same time, you and all your audience and everyone who's in this project of writing and reading things together can decide, "Okay, but it's only going to matter if we wrote it," and we're going to make this choice. And it matters because it's hard. It's not easy for me to write this thing, and it would be super easy for me—and other people would know it would be easy for me—if I just prompted the machine and then output the machine thing. So the value comes from the fact that I spent a part of my short and precious life to write this thing for you, even if by some eval, it's worse. And then you've created meaning. You've created real work that matters as well. And I don't think it matters any less just because all the humans involved are eating food grown by robots or something like that. Because they've all decided, and that's fine. That's not what we care about.

Jackson: (1:40:40) We get to make our meaning. We get to choose.

Zhengdong: (1:40:42) Yeah. And maybe later on, at a different level of abstraction farther into the future, it's like, "Damn, I'm going to send my starships out to these galaxies." And the work is hard to click this button or something like that.

Jackson: (1:41:00) You're playing Starcraft? Just a handful of additional miscellanea. One of my favorite parts of that recent letter is you talking about pluralism, marginalia, and optimism. I think it's worth people go reading it, but one of the parts that really anchors it so well is Andor. In Season 2 of Andor—I love Andor, but I'll open up to you. Why are those three values so highly prized to you?

Zhengdong: (1:41:28) Yeah, I think maybe pluralism is just the most highly prized value to me, even though I recognize the contradiction of, like, "Oh, what do you mean the most highly prized value is pluralism?" or something like that. The other two I just wrote as sort of themes to personal life of the year, where I got to travel to a lot of different countries and I got to see a lot of tiny bits of bureaucracy, and then I got to see a lot of people being so optimistic. And then for that letter, I wrote about Isaiah Berlin a lot. I really imagine him as both an intellectual role model, where I think I agree with him about a lot of what he says about pluralism—not being able to rank order your values, as a lot of philosophies related to AI are very prone to doing, the hedgehog and the fox, as you know—but also a sort of role model as a consumer-producer. In his life, he basically knew everybody who was alive at the time, or got to talk to them, and lived this very full—

Jackson: (1:42:43) He was a good hang.

Zhengdong: (1:42:44) Yeah, exactly. And he also produced stuff and he also spoke to the general public. A lot of his lectures or his essays were very helpful to those outside academia, just explaining ideology in the 20th century. And one of my aspirations will be to just do a tiny bit to help explain technology, which I think is the thing of the 21st century. It probably has always been. But he is also a role model in that way. So talking about Berlin a lot, pluralism being an important value—and the other two are mainly just things that I felt like tied together the other stuff that happened in the—

Jackson: (1:43:30) Marginalia came up even in our conversation. It's the list of names, the people on the edges. What idea of Berlin's or piece of Berlin's do you think would be most impactful for people to know about or familiarize themselves with? Would it be The Hedgehog and the Fox, something else, or maybe not a piece but just an idea?

Zhengdong: (1:43:48) Yeah, I think just the idea of the hedgehog and the fox I found really fun. And people have mentioned there's just something about animals, you know, imagining the animals.

Jackson: (1:43:59) I liked your little short story, by the way.

Zhengdong: (1:44:01) Oh, thanks. Yeah, so I think even just the idea and reading a few pages of the essay. It's quite a long essay and most of it is about Tolstoy, actually, so I found it super interesting. But yeah, it is maybe a bit separate from just the idea where he wants to write an essay about Tolstoy, but he's like, "Okay, but how do I introduce this essay? Tolstoy doesn't fit in these two categories perfectly in the way that all these other people in history do. Let's just begin with this thought experiment: This is what a hedgehog is. This is what a fox is. Who's a hedgehog? Who's a fox? So as he wrote, a hedgehog knows one big thing and a fox knows many things. You can think of it as a generalist versus someone who is obsessed with something. And then if you take in some kind of consilience, value, pluralism, everyone should really be both at the same time. Always.

Jackson: (1:44:58) It's always both. Just to wonder, optimistically, briefly, for a moment: how might things be if all of this really works? How might things be really great? What are you excited about?

Zhengdong: (1:45:17) Yeah, I just think that—also to connect it to things not really being different because they're different all the time—I think that in the wealthy world, we basically have already reached this. People who are very wealthy in the world today—probably a lot of the people listening to this—never have to worry about food or water. They have a lot of choice in what they can do with their time, and have a lot of choice in how they decide their own identity and what they value. Of course there are limitations, and as humans, maybe we focus too much on them—like, "I have to do this; I really wish I didn't have to do that." I think that this will become available to a lot more people. We have to deal with inequality as an issue, but making everybody wealthier works when making people wealthier is basically free. Not only that, but second, people who were already able to do this will come face to face with the fact that you should be doing this earlier—you should be doing this now. You could have always been doing this. The thing about AI as a thing—just like the pandemic as a thing, or all these serious kind of near-term thought experiments you could do about your own life—will really make you come face to face with it. Maybe people already are, and that's why some people turn to religion and things like that. I think we will just be engaging with this question a lot more. I also really am inspired by Demis; maybe we will really find the true nature of reality in some way that's very convincing to us as well.

(1:47:13) Annual Letters, Economist Obituaries, London, and Burke

Jackson: (1:47:14) You write annual letters largely inspired, I think, by Dan Wang.

Zhengdong: (1:47:18) Yes.

Jackson: (1:47:20) There was a bit where you're quoting Dan advocating for the letters. First Dan says, "I don't understand why more people aren't writing them. It's not just about sharing your thoughts and recommendations with the rest of the world. Having this vessel that you're motivated to fill encourages being more observant and analytical in daily life too." And then you say, "He's right. This letter, the only deadline I give myself every year, is an immensely powerful nudge to do more interesting things during the year. If only subconsciously, it's the best antidote to the temptation to time-box research, writing, or enjoying life." How has writing these letters changed you? How has it improved your life?

Zhengdong: (1:47:57) Yeah, I think really a lot of it is subconscious where you think, "Okay, life is long or short," however much you say it. When you think of a whole year—and I don't know if people usually think in the future in 12 years or five years.

Jackson: (1:48:17) You wrote somewhere, by the way, about a 12-year plan.

Zhengdong: (1:48:19) Yeah, because it's so divisible. In the same way that you can't really plan ahead, therefore you should really plan ahead. This is a tangent, but because the near future is so uncertain, then if you were to think, "Who am I going to be 12 years ahead?" you really focus on what doesn't change, or what do I really value. That sort of thing is very clarifying. Subconsciously, I think through the year you just think, "Oh, this will be good for the bit."

Jackson: (1:48:56) Are you writing throughout the year?

Zhengdong: (1:48:58) No. I have a big Apple Note that I just put bullets in. These could be thoughts, or I have to probably fit in this important event in AI at some point just to not forget it. The model releases are so fast and frequent, I'm like, "Did that model come out this year or last year?" It's just one big Apple Note.

Jackson: (1:49:22) Is there anyone you really wish would write an annual letter?

Zhengdong: (1:49:26) I think all my friends should write one.

Jackson: (1:49:28) Yeah, it's hard. These are like 12,000 words, but you—

Zhengdong: (1:49:31) You only have to do it once a year. I don't think I'm very productive as a person. There are people who are churning out 3,000 words a week. I think it's harder when it needs to be some kind of synthesis or you're reflecting on something instead of maybe just journaling. There are no set boundaries of how far back you have to go or what you have to cover. But it's easier in that it is personal. It just so happens that personal stuff is scarce, and personal stuff is what AI will never be able to automate. Writing about personal stuff is a lot easier, I think, than having to reinvent philosophy and come up with a new theory.

Jackson: (1:50:15) You do a little bit of both. But how have you become a better writer over the course of these? You've done four of them now.

Zhengdong: (1:50:21) Yeah. Every year I write these, I also keep a little list of things I have to remember about writing the next time around. Some of them are just things that everyone knows but are still really hard to do—like just getting words on the page is the most important thing, and you have to get through the bad words to get to the good words. In the same way in the rest of my life, I rewrite lists all the time. It's sort of like you have a row of your ducks and you're patting all the ducks—"Yes, you're still there"—and I've done the next sequential thing on the list. I think that often rewriting end-to-end is helpful. Even if you're writing the exact same words, you're reading it as a reader would read it. By the time you're at the end of writing a longer piece and you've spent a lot of time with it, you're lost in the sauce. For my last letter, just the opening few paragraphs of how to try to AI pill somebody—it doesn't work on me anymore because I've read it so many times. But maybe you can salvage that a little bit by starting from a blank page and writing sequentially exactly as the reader would read it. In a very short context, you load it all into the memory: "Okay, so the reader knows this fact now, and now I've introduced this proper noun." I'm loading the reader's context as they would read it. Then I would get to a point of, "Oh, this doesn't make any sense because I'm lost in the sauce," and I've mentioned this concept before even introducing it.

Jackson: (1:52:08) Wow, that sounds exhausting, but probably good advice. Probably effective.

Zhengdong: (1:52:12) It doesn't take that long, actually. I think, at least for me, the main bottleneck of writing is just: am I writing directionally correct? All the time that I spent retyping the exact same words is not the bottleneck, and it can also add some momentum.

Jackson: (1:52:34) Get the pedals going. Why do you love The Economist's obituaries?

Zhengdong: (1:52:38) So one is there's only 50 a year. Two is they pick subjects that are not just famous people. So if the Queen dies, the Queen gets an obituary, but a lot of them are people most people will not have heard of. Someone who was just a really important member of a community—like this tiny island off the coast of Scotland, and there's this guy who's really important to that community, for example—or the last speaker of a language, or someone who started a school for disadvantaged children in a particular part of London. I just love learning about this or acquiring this new information that I never would have known otherwise. A lot of it is curation; I think the curation is excellent. The fact that they're written from the perspective of the person in a way that's more—it's not fictional, but there's more flourishes that are associated with trying to put you physically in the place. Maybe there wasn't an ocean breeze, but who does it hurt to pretend there was a breeze on this important day in this person's life? All Economist pieces are quite concise, and so it's just wonderful.

Jackson: (1:54:05) One or two that you would very specifically recommend come to mind?

Zhengdong: (1:54:11) Yes. Okay, so Pasha Lee and Albert Woodfox, both from 2022. And then I also want to say from that year, Thich Nhat Hanh. I emailed Ann Wroe, the editor, saying she should do an obituary for him, and she replied just like, "I'm on it." So I don't know if I actually made a difference, but it would be quite cool.

Jackson: (1:54:36) I don't know—correlation, causation. That's pretty cool. Wow. One Hundred Years of Solitude. Why is it your favorite?

Zhengdong: (1:54:43) It's just really beautiful. And I think it's a book about everything, so everything is in it. And maybe even more than 100% of everything is in it because you've got people just suddenly floating off to heaven and gypsies visiting your village, showing off new technologies that never existed. I think there's a way of doing magical realism or trying to write a book about everything that just doesn't work. And this maybe doesn't answer your question, but I just feel like everything fits together so well. Or nothing feels out of place, or nothing feels like this is just an extraneous detail or something. Somehow everything just fits in super well. Generally in books and films, I like ensemble casts or many generations. It's just so rich.

Jackson: (1:55:37) Have you ever seen Magnolia?

Zhengdong: (1:55:38) Yes, I love that one as well.

Jackson: (1:55:42) A book I love, When We Cease to Understand the World. This is you: "I won't presume to tell you what you should think after reading this book, but surely everyone who reads this book will agree. Any scientist who reads this book and also thinks their work is worth a damn should think something." I don't have a question. I just wanted to read that. That was good. Has that book—I assume you've read The Maniac as well?

Zhengdong: (1:56:05) Yes.

Jackson: (1:56:05) I still haven't. But I'm curious which book feels more resonant for the time. Is it still—

Zhengdong: (1:56:14) I think When We Cease to Understand the World is more evergreen. And then for me, the last third of The Maniac, which was about DeepMind and AlphaGo—I think it goes over a lot of the same ground of the documentary. So watching the documentary is good. In general, I just think the author, Benjamín Labatut—he's apparently friends with Demis Hassabis. And I think he is also really good at keeping in mind the sort of weirdness or the grandness of the scale. He would also be someone I would look towards whenever I'm like, "Oh, is all we have really this techno-singularity where at the end of it we get flying cars?" No, we can be much more ambitious than that. We can face the whole scale of all we don't know in the universe, and then that's really maybe the most ambitious we could be.

Jackson: (1:57:18) There's a quote from him, I think in the interview with Jasmine that I'm sure I'm going to butcher, but it's something along the lines of: "That's the thing about humans: we're far better at being than we are at knowing."

Zhengdong: (1:57:34) Yeah, everyone should watch that interview with Jasmine.

Jackson: (1:57:37) Really good.

Zhengdong: (1:57:37) Yeah, really good.

Jackson: (1:57:39) We're in London. You seem to have a lot of love for this place. You've called it the best pre-AGI city and the best post-AGI city.

Zhengdong: (1:57:46) That's right.

Jackson: (1:57:47) What do you love about it?

Zhengdong: (1:57:51) Other than the long litany of practical things like parks, best airport—I will not be elaborating—and food, I just think it's a great representation of pluralism. The fact that there have been 697 Lord Mayors of London, or something like that—that's existed for a long time. There must be something about the city existing for that long, having such a diversity that anything you want to do, you would be able to find a scene for it. The fact that people are so reasonable and so funny—it's great.

Jackson: (1:58:42) A good place to be a consumer.

Zhengdong: (1:58:43) Yes, for sure. The best place.

Jackson: (1:58:45) Do you think you'll ever make a game?

Zhengdong: (1:58:47) Yeah, I think so. And it's getting easier every year. In terms of whether I should take a break or something, yeah, I've always wanted to make a Chinese history-inspired Game of Thrones kind of thing. I think people should write a series like this too. I've tried Ken Liu's Dandelion Dynasty. Unfortunately, I couldn't get into it as much, but I think there's a lot of room for that.

Jackson: (1:59:17) What about container ships? Why are you so into them? Maybe you're not so into them, but you're into...

Zhengdong: (1:59:24) I am so into them. They're just so efficient. We don't wish for world government or anything, but the fact that everybody has agreed on this kind of standardization—I think the benefits of how much it's improved our lives is just hard to comprehend. You know how people ship trash to a different country to be sorted and then shipped back? That turns out to be the most efficient way to do it, maybe.

Jackson: (1:59:59) Okay, so there are a few ways that top-down total control, or at least total collective decision-making, can be pretty good.

Zhengdong: (2:00:07) I think just agreeing on some standards, which doesn't seem like it should be that high-stakes, could have a lot of benefits.

Jackson: (2:00:18) Yeah. I think I referenced it earlier. You often cite this Nabeel Qureshi and Dan Wang kind of advice on productivity, which is just "do more." I think you say you can have a free lunch across the Pareto front. You also say somewhere else—forgive me because I don't have the date written down, so it's possible this was two years ago, but this is in one of the letters: "The problem is more general than exercise, though. If I want to read, play music, practice Chinese, and pick up new hobbies and it isn't happening more by now, what makes me think it has a better chance of happening later? It's time to either change or quit. I was optimistic last year about a big virtuous cycle where doing everything makes everything else easier. I'm going to take the opposite view this year that I should be honest about the actual trade-offs I face." Either way, it sounds trite, but I'll figure it out one day. Obviously, I don't think these things are fully mutually exclusive, but I think one of the things we gradually learn—sometimes it hits you in the face—is that there isn't that much time. And there you really do have to choose. I think the "doing more" thing is also true, but I was just curious how that's going.

Zhengdong: (2:01:21) Yeah. I think the year after that, I sort of even said slightly the opposite, where I'm going to do more. So, yeah, like a lot of things, like when we started off talking about research, you take a position that is maybe slightly too much to the extreme and you learn something about it, and then you just develop a better taste or prioritization so that in the future you are doing the balance better.

Jackson: (2:01:59) Yeah. Do you ever let your computers idle overnight?

Zhengdong: (2:02:02) I think this depends on compute allocation for the team where it's shared across the team. So I rest assured that the computers are never idle.

Jackson: (2:02:14) Well, I feel like including many non-AI researchers, most of the people I know are paranoid and freaked out to ever leave Claude not running on their computer.

Zhengdong: (2:02:22) Yeah, but that's also no way to live. I think there are bigger costs to that where, yes, you're letting your computer or Claude idle, but if your production function—which I really think all humans are—just needs time to think, one should not be concerning themselves with tiny things like optimizing their usage so that you can focus on bigger problems. Maybe you should hire somebody to use your quota efficiently, but you just need to be focused on the big problems. There will be bigger costs that come with it if you micromanage your quota and your time.

Jackson: (2:03:09) Maybe leave the optimization to the machines.

Zhengdong: (2:03:12) Sounds good.

Jackson: (2:03:16) Worth shouting her out because I'm having dinner with her tonight, but I know you love it. Why do you resonate so much with Everything's a Scam?

Zhengdong: (2:03:26) Yeah, I think it's just a great reminder. Maybe in the same kind of thing where every time you read a work of fiction, say Foundation, and it just makes one point super, super well—this is a great reminder. A lot of the things Riva-Tez does are a great reminder. The fact that she made this a song and it's on Spotify, it's very much "you can just do things." Everything's a Scam opens up your ability to do things. I'm also very inspired by her opening a toy store in London and a lot of other schemes that she's running. "Scam" isn't so bad. It's just that the rules are changeable, or the rules are made up and not totally set in stone.

Jackson: (2:04:21) Indeed. My last thing is a quote from Burke that you quote in one of your letters. This felt to me fitting because it seems like a case for the long journey of research and discovery and adventure. I'm quoting now: "By a slow but well sustained progress, the effect of each step is watched, the good or ill success. The first gives light to us in the second. And so from light to light we are conducted with safety. Through the whole series we compensate, we reconcile, we balance. We are enabled to unite in a consistent whole the various anomalies and contending principles that are found in the minds and affairs of men. From hence arises not an excellence in simplicity, but one far superior: an excellence in composition."

Zhengdong: (2:05:08) I think it just encapsulates the point I was trying to make there, but also what we talked about of trying to make the transition to powerful AI go well. There is just a lot of talk about this time is different, about step changes, about distinction, right? And that's very useful in shaking people and getting them to feel the wave crashing over them. But once you get there — or sort of be careful what you wish for — maybe you don't want the prize on offer. And really, the more effective or the better way to do it is there are these promises that we've made before and we're going to keep them. The important stuff is not going to change; the important stuff has always been there. AI just makes you face the questions that you should have been facing all along a lot sooner, or at the right time. I think that it's great that something that Burke wrote so long ago is still relevant and a great point for him as well.

Jackson: (2:06:25) Indeed. Anything else you want to talk about?

Zhengdong: (2:06:27) No, that's good.

Jackson: (2:06:29) That's all I got. Zhengdong, thank you so much. This was wonderful.

Zhengdong: (2:06:31) Thank you. It's been wonderful.

← Back to all episodes