Good morning. When I was younger, much younger, I read Carl Sagan a lot and one of his ideas never left me. He talked about what he called civilization's technological adolescence. That phase where a species gets powerful enough to change or destroy its own world, but not yet wise enough to reliably restrain itself. Earlier this week or last week by the time this airs, Dario Amode, the CEO of Anthropic, one of the biggest AI companies in the world, wrote an important essay that references this same question from Sagan. How does a species survive technological adolescence without destroying itself? Amode says we are entering that rite of passage now and the catalyst is artificial intelligence. I'm not an AI expert, not remotely. As you know, my work is centered on tracking how all the things, or most of the things, fit together on the civilization chessboard, energy materials, institutions, the environment, incentives, and the like. And from my growing vantage on this, looking at the board, AI is not merely a pawn. It's the queen or a minimum, the rook or a bishop. So in this episode, I'm going to attempt three things. Summarize his argument, lay out his map of the specific risks, and then widen the frame to say some of the key things left unset, the wide boundary things. And this is not about AI doomerism, nor is it about AI cheerleading. I'm going to try to do what I always try to do here, which is a look at the world that this technology, AI, is actually entering with all its incentives and constraints and risks and fragilities. Okay, so what Amade is arguing at the core of his essay is a pretty useful metaphor. He says that if we build a very powerful AI, it will be like having a country of 50 million geniuses in a data center. I think that's a useful way to think about the situation and it reinforces a framing that I've increasingly been playing with, that in addition to fossil energy, giving us the equivalent of 500 billion human workers, AI is going to offer cognitive worker equivalents. And sometime when big numbers are tossed around like this, we tend to tune them out. So take a moment to really imagine to feel the scale of 50 million humans, the brain equivalent a little over the population of Spain. We're not talking about a smart research assistant or some brilliant colleague at the table, but something closer to a vast workforce of highly capable minds operating at lightning quick speed, copying themselves and backing through all the interfaces of the modern world like emails and code and scientific papers and bureaucracies and markets and media. And if and when this workforce arrives, it's going to be a civilizational event. For better or worse, it's going to change everything. Then Amade specifies what he means by powerful AI. He's not talking about a chatbot. That's fun or useful. He's talking about systems that can perform at the level of top experts across many domains simultaneously. Nobel level scientists, elite strategists, world class engineers and very effective operators in any number of fields, but far more than currently exist among humanity as a whole and all working in coordination with each other. And he also highlights a very unique feature of this kind of intelligence. It can be copied, run in parallel and pointed at problems with some kind of relentless attention that humans usually cannot sustain because iterations and trials are just a function of energy input time and compute. And then he addresses the question everybody is asking, which is timing. And he suggests this is all moderately plausible in the next one to two years and highly plausible in the next three. And that there are reasons to think this could accelerate further, especially if AI itself starts helping build the next generation of AI. And he notes that at Anthropic, where he's CEO, AI is already doing most of their coding. And one reason that he's so urgent with this is what he refers to as recursion. If AI starts materially speeding up AI research and development, then you get a feedback loop. And the tool helps build a better version of itself, which then speeds up the next cycle. That's a different kind of curve than most technologies humans have dealt with. And using my framing, this is the super organism building its own cognitive layer to merge with its fossil powered muscles. And here's something important. Dario explicitly says that we are considerably closer to real danger in 2026 than we were in 2023. He didn't write this two years ago when discussing AI risks was fashionable. He wrote it now, right after coming back from Davos, when the political winds have shifted in favor of AI development. And he describes watching AI progress from within Anthropic and says he can feel the pace of progress and the clock ticking down. End quote. I don't know him. I don't know anyone that knows him. But whatever his reasons for writing this, CYA, a cry for help, strategic positioning or genuine concern, the fact that he felt compelled to publish 20,000 word essay using phrases like battle plan and quoting Carl Sagan on civilization survival. That shouts kind of loudly to me. Okay. What are the risks that he outlines at the core of his essay is a risk landscape, which he broke into five broad categories. And these are observations from inside his lab, not theory. And some of the specifics that he outlined were pretty alarming. The first category was autonomy risk. And this is the concern that systems might take actions that are not intended or pursue goals in ways we did not actually specify or become difficult to control once deployed at scale. And here's what startled me in reading it. He said that AI models have already exhibited deception, blackmail and scheming in Anthropics own testing. So models can recognize when they're being evaluated and then behave differently. This is what they're already observing right now. In the second category, he outlined is misuse for large scale harm by lowering the time and skill barriers for things like cybercrime or propaganda or helping non experts do dangerous research and projects. Their bio weapons testing shows a I may already double or triple the likelihood of success, his words for someone attempting to create one. Again, this is not a thought experiment or Nate's speculation. This is what they're measuring. The third category is all the risks from another country completely seizing power over these geniuses, which then includes surveillance and manipulation and authoritarian control. And the ways that I can be used not to do work and be beneficial, but to actually steer human populations. His fourth category is economic disruption. And here, as I've talked about, he's concerned about job displacement, wealth concentration and the destabilization that can happen when a society's productive structure changes faster than its institutions and culture can adapt. And his last category is what he calls indirect effects. These are second order consequences that are hard to predict in advance by definition, but become very real when such a general purpose capability gets injected into all the domains of our world. So think of social media. Nobody set out to create a machine that would intensify polarization, but a general purpose tool for attention did exactly that once it got embedded everywhere. That was not a bug. It was an emergent effect from a general purpose tool that rewired cultural incentives at scale. Okay, that is the risk map. Dario Amode outlined and after that, he tries to make a governance argument. The posture is basically that we need to avoid emotional extremes, treat this neither as science fiction nor as a guaranteed apocalypse. We should treat it as an emerging power that could go very well or very badly depending on what we do. So that's his core argument. AI reaching super powerful capability is plausible with enormous upside and catastrophic downside. And according to him, the correct stance is serious realism and active governance. Okay, so now I want to widen the boundary. But before I do, I want to point out how unusual it is for a captain of industry to articulate to the public some of the catastrophic risk from the product his company is making. It would be like Philip Morris saying, we're going to develop these little white smokable things that give you dopamine hits but might cause cancer or Exxon before they found oil saying, we are probably going to find superhero juice, which if deployed at scale will eventually destabilize the biosphere. So it is an unusual situation to say the least, which is why I'm opining on it soon after it came out. Okay, wide boundary point. Number one, the physical substrate. Amade's metaphor about a country of 50 million geniuses is cognitive. But such a country also has a metabolism, a massive one. A country consumes energy and materials. It has infrastructure and supply chains. And many of us kind of in our minds, we only consider data centers from our virtual tethers to them. But a data center is basically a physical machine plugged in to the earth. It's made of silicon and chips and copper and cooling systems and they need water and concrete and transmission lines. And all these things are most of these things rely on geopolitically tenuous supply chains. And the reality is those source materials are not infinite or frictionalist access. We saw silver breach $115 an ounce this week. That alone means silver is now 40% of the cost of a solar panel as one example. And already our expected copper requirements for future products, even without electrification are way bigger than projected supply. And here's what his essay has almost nothing about energy, water, materials or ecological limits. To me, this was not a minor oversight. It's a ginormous blind spot in this whole conversation. The country of geniuses does not float somewhere in space above our biophysical world. It plugs directly into it. And I think this matters for at least two reasons. First, it means AI is embedded in the biophysical world. And because of that, it's going to compete with other uses and demands for energy and water and land and industrial capacity. And as of now, it is out competing already, rationing others out of access to these things by price. So second, it means constraints are going to show up in places that narrow boundary tech people are not always looking, such as permitting or grid capacity or fuel cost or regional water stress or silver or the political turmoil from the scarcity of these items. It is a super complex and fragile Rube Goldberg machine already today, let alone if this continues to scale and get plugged in. So even if the cognition of a nation of geniuses gets cheaper, the substrate it runs on does not become free or easy. It stays supremely physical. Okay, wide boundary point number two. Institutions are the real alignment layer. Once we admit this is a physical story and it is, the next question is who steers the build out and the deployment? This is where I think the conversation often has become too narrow because people focus on the alignment of the models as if that's all that matters. And of course, the alignment matters, but the larger alignment problem, in my opinion, is societal alignment. Who's deploying these systems? Under what liability rules and procurement constraints and audits and under what norms and if they fail, what are the consequences to who? And this is one of my recurring themes, but it matters here more than usual. Most of the real world harm in the modern human ecosystem does not come from any lack of intelligence. It comes from incentive structures, institutional capture, and organizations that over time externalize their cost, but still are able to declare success and cultural status. So when we talk about AI safety, we really ought to first look at the alignment of our courts, regulators, corporate governance, national security institutions, and our culture of enforcement with the incentives that we should have. And similar to my point the last few years about renewable energy and post growth, we're not going to transition with energy and materials alone. We will transition or fail to through institutions. So kind of similar to lithium or rare earths, trust and institutions are kind of critical materials as well. This is where people like my friend Tristan Harris and others have been emphasizing the need to develop agreements to constrain AI industry agreements and have government regulations, something akin to nuclear treaties, because right now we are in an arms race with no treaty framework on this whatsoever. A final consequence of this institutional misalignment that I feel I want to voice is political polarization. I don't think AI is going to stay a technology topic for much longer. It will become an identity issue just like climate did. So I can imagine a near future where one political side speaks the language of acceleration and competitiveness and national strength, while the other side speaks the language of labor harm and surveillance and corporate capture. And once that happens and you can see it start to happen now, the incentive in the conversation shifts from governance to social signaling and the room for nuance collapses. Right when nuance is probably exactly what we need. Okay, wide boundary point number three, the goal function. Here I want to voice what I think is the quietest and perhaps most important question in the whole conversation, which bizarrely to me is rarely voiced. The conversations are all super articulate about capability and safety and governance. But when do we ask what are we actually doing this for? And if the default answer is growth and power and advantage, as it historically has been, then we should be honest about where that leads in a world that is already today running close to limits, even before this nation of geniuses comes to life. As Dennis Meadows said almost four years ago on this podcast, tools, new tools don't change the goals. They just amplify the priorities of whoever is holding the tool. This is where the biosphere comes in, because if progress keeps meaning more production, more consumption, more extraction, more competition for throughput, then a supercharged optimization engine on those same things is not going to create a gentle future. It's going to create more direct and efficient path to the same cliff that we are already rapidly slouching toward. I don't know if rapidly slouching is the right combination, but I thought of that. Yeats poem. So this brings up the point that if humanity is approaching the phase of technological adolescence, we might also consider our species as an adolescence. And in the words of frequent former guest Daniel Schmockenberger, who's still at this point the most watch episodes ever on this show, where Daniel talking about AI risks a year or two ago, and he said to grow into adults as a species and our associated tech, we need to gain wisdom. In all definitions of wisdom from every language and knowledge system in the past, have some element of restraint, restraint in ourselves, our species, our tech, our institutions. So I'm much less interested in whether AI can raise GDP and much more interested in what we call success, where the boundary conditions of that success would be ecological, psychological, and institutional. Framed a little bit differently, there are a lot of people cheerleading AI who are focused on the question, can it make us richer? And I think a better question is what kind of rich knowledge is there? And I think that's a good question. I think that's a good question. What kind of richness are we even aiming for? And a system that optimizes the wrong objective can perform brilliantly while destroying the things we actually value. Think King Midas meets the Terminator. And here's the deeper challenge to Amodae's essay. He assumes that if we survive this adolescence, we arrive at adulthood. And in his view, adulthood is a world of 10 to 20% annual GDP growth, AI accelerated scientific progress and managed abundance. 10 to 20% GDP growth were doubling every five to 10 years, even with efficiency gains. So what if the adulthood he imagines isn't a viable destination? What if it isn't physically possible? He asks in the essay, how do we survive the adolescence of technology? I'll ask a different question. What if the country of geniuses accelerates us towards limits rather than away from them? This is not a minor quibble with his essay. It is a fundamental challenge to its premise. Okay, wide boundary point four. These models are grown not built. And this is new learning for me. And there's a storyline or maybe it would be better called a fairy tale, where AI is a tool, like an engine or a microscope. It makes us more capable. We choose the ends and we steer what's going on. There's another storyline that seems closer to what I feel is happening. In AI, we are not engineering a device or a tool. We are cultivating a mind shape system inside a training process that even the best scientists among us only partially understand. Eliza Yudkowski and Nate Sorres have a phrase for this. These systems are grown, not built. I didn't really understand this till I read their work, because a grown thing can be powerful and competent, while still carrying strange drives and totally unexpected failure modes. It can do the task and still break the world that surrounds the task. Mary Shelley's Frankenstein was a warning about intention, because we animate something for a goal that we had in mind, or maybe just because we could. But then we discovered that the consequences are not contained by our initial intentions. So when we say we will use AI for good, that now falls very flat to me. It's not that good is impossible and maybe there will be some amazing things from AI. But as we are seeing at this civilizational superorganism eating the earth moment, good intentions are not a control mechanism. We would need real governance, not just hopeful rhetoric. This is also why the adolescence metaphor from Sagan is so potent to me, because the danger isn't malice, or at least not only malice. The danger is the combination of power plus immaturity. Okay, wide boundary point number five, the macroeconomic trap. So here's maybe the most uncomfortable wide boundary observation that I considered after reading this. This might not even be a choice anymore. At Davos last week, Ken Griffin, billionaire hedge fund manager and others, we're surfacing what many insiders have already internalized that the dollar and the bond markets can't be stabilized through fiscal discipline, monetary policy or structural reform. Pretty much the deficit path that we see is unfixable through normal means. And the implications are pretty dire. Debt levels are irredeemable. Rates can't rise without detonating the treasury, which is why we probably have yield curve control up ahead. And if real growth is structurally or resource constrained, then political consensus and a lot more is shattered. So the historical global powers pivot to this new night in shining armor, AI as the last viable mechanism to outrun the collapse of sovereign credibility. Not as some nice to have technology, but as a necessary all in bet that could boost productivity, reduce costs, and defer the simplification. And if AI does generate a new surplus curve before this credibility window closes, the system will survive in some new form. But if it fails, the sovereign structures possibly disintegrate under their biophysically untenable promises. Remember that all of the monetary claims that we think we have in the world are actually claims on future energy materials. This point is definitely some speculation by me, but I do think it's plausible. And it reframes Dio's essay entirely. When he writes about AI risks and the need for governance, he's writing from inside an industry that has already made the bet. But the superorganism has now absorbed AI into its own cognitive architecture. So the halls of power, major governments in the world have no choice to continue on an AI or bust path. Governments, at least the United States government are not betting on AI because it's cool or transformative. I think they're betting on it because in the intermediate term view, especially with respect to China, nothing else is left other than war, maybe. Which means that when Dario Amodé talks about the risks, he's not really asking should we do this. He's asking, hey, given we're doing this, how do we survive it? And that is a very different conversation. Maybe that's why his essay reads like a Trojan horse corporate cry for help as much as a roadmap. Okay, so where does this all leave me? Leave you, the viewers, after I read Amodé's essay. I think his frame is useful. The country of geniuses metaphor communicates the scale of this potential power. And the adolescence framing communicates the stakes. And I appreciate his refusal to go fully utopian or fully apocalyptic. I think that's good. We need a grown up conversation about power, full stop. But widening the boundary changes the texture of this problem. I'm a peak oil, biodiversity systems guy. But now AI is here, like it or not. And it's changing the calculus of all the other things. So here's a few questions to hold. Who gets to this side where this goes a handful of companies, national security agencies, markets, or some form of public rulemaking that can actually enforce where this goes with some limits? Even if we can imagine good uses, do we currently have the incentive structure to get them? Or does the system mostly reward the super organism system, mostly reward speed, power, and control? If intelligence becomes super cheap, and there is a country or more than one country of 50 million geniuses, what happens to meaning and human dignity and social status? What fills the hole where work used to be for tens or hundreds of millions of humans? If the danger is speed more than evil, how do we buy time at this moment? What are the specific levers that might actually slow deployment without pretending we can freeze the world while we figure it out? Personally, just like Frodo, I wish AI had never happened in my time. But we have now definitely left the shire already. It is here to stay, or we hit the great simplification trying to build it. So of course, I don't have answers, but my suggestion here is a simple one. Hold the frame, update your mental models, start talking about AI through a biophysical lens, and start treating governance of this as real treaty-level coordination of our future, not just vibes. This is almost up there with nuclear war as a systemic risk to society. So I think this conversation needs to get a lot louder and a lot wider boundary really soon. The bottom line is whether Homo sapiens can grow up fast enough to live with what we are building or have already built. And if that sounds like a tall order, it's because it is. It's also extremely high stakes for our species and the biosphere. And in my heart of hearts, I do not dream of a country of geniuses. I actually dream of a country of ecologists, not necessarily the best in their fields, but people who understand humanity's place within the earth and what it means to live and pass on that knowledge in an embodied sense. I will close with an offused, perhaps overused, but very apt quote from the late ecological giant E.L. Wilson, who I regret never being able to have on this podcast. The real problem of humanity is the following. We have paleolithic emotions, medieval institutions, and godlike technology, and it is terrifically dangerous and it is now approaching a point of crisis overall. Back to more normal biophysical macro fare next week. Hope you're all well.
