Gradually, then suddenly
i am a canary
A few things I want to cover:
Why current economic conditions are especially problematic from an AI perspective. Plus a brief comment on the potential use of AI in creating tariff policy.
How incremental improvements actually make it seem like progress is slower than it is.
A recent forecast called AI-2027 and the related ideas of the intelligence explosion and legibility of the universe.
I am increasingly concerned that good outcomes are less likely and the amount of luck required for things to work out well is uncomfortably high.
Current Economic Conditions and AI
I’m going to mostly stay in my lane here and not provide much commentary1 on current economic conditions and “policy”2 directly, but I do have perspective worth sharing about the relationship of these changes and the risk they pose due to AI.
My main concern and contention is that increasing economic volatility will make at least a subset of firms more risk seeking. And firms that are more risk seeking will make bigger bets on AI. There are at least two ways in which this will probably be bad.
First is that this could substantially exacerbate layoffs and hiring reductions at a time when this might already be quite problematic. Overall disruption to the labor market continues to be one of my primary concerns around the potential negative impacts of AI and in a highly volatile environment, this could just be accelerated.
Second is that this could potentially go quite poorly. Things are changing rapidly, but trying to make an epic shift to AI even 3 to 6 months too early could be disastrous. But in desperation, firms may make decisions they would not otherwise do and over-commit before the technology is ready to justify the switch.
Something that is quite likely bad for both the individual humans being laid off AND for the firms making these decisions is a classic lose-lose and it’s just hard for me to imagine this not happening. The current situation cannot be responsibly handled by business as usual. One of the main protective layers slowing down changes in this domain has been my belief in the glacial speed of organizational change. But leaders respond to incentives and disastrous loss to projected earnings will inject some adrenaline into the system.
Business decisions around AI were always going to be made under a lot of uncertainty and incomplete information, but what’s new here is are the additional elements of decreased time, increased pressure and even more unpredictability.
So basically, my take here is that, when viewed through the lens of AI, all of this (gestures broadly) is somehow almost comically worse.
Emotional status: increasingly unsettled by uncertainty.
AI use in tariff policy?
I do also feel compelled to briefly comment on the potential use of AI in formulating the tariffs because I feel like much of the commentary here is missing an absolutely critical point — even folks like Paul Krugman on Ezra Klein’s podcast. The common narrative I’ve seen here is that AI suggested this simplistic and naive approach. But this overlooks some critical text that the models did in fact produce (I’m referring to this tweet — all of these bullets are directly pulled from those model responses):
This method ignores the intricate dynamics of international trade -- such as elasticities, retaliatory measures, and supply chain nuances--but it provides a blunt, proportional rule to “level the playing field”.
While this approach is straightforward and mathematically simple, its critically oversimplified.
However, I should note that economists generally caution that trade deficits alone don't necessarily indicate unfair trade practices, and tariffs can have complex economic consequences including increased consumer prices and potential retaliation.
Would you like me to elaborate on any part of this approach or discussion potential refinements to make it more sophisticated?
While the goal of balancing trade is understandable, devising an "easy way" to calculate tariffs to achieve this is extremely complex and most economists would argue its not feasible or desirable for several reasons.
However, let’s outline a highly simplified conceptual approach based only on the numbers, ignoring the vast real-world complexities and consequences.
The models answered what was asked for. But they also strongly caveated the information they were providing. My contention here is that if somehow AI models were used in this process, they were used with such absurd incompetence that a high schooler trying to cheat on an exam would be embarrassed.
The reason I think this matters so much is that the common narrative suggests that the AI models somehow endorsed or recommended these tariffs in this way, when quite to the contrary, they strongly discouraged it. And this isn’t even with any sophisticated prompting or iteration. I’m defensive of the AI results in this case because the common narrative dramatically underestimates the model capabilities which I believe is problematic for us all.
Incremental progress is a lullaby
Gemini 2.5 was released3. And, no one seemed to care that much. It’s probably the best model publicly available. But it’s also just a little bit better at a few things and much of that doesn’t matter much to people.
Even if you don’t care about its other improvements, the super power I want to pitch to you is its ability to summarize and engage with Youtube videos. You can just provide a link to a Youtube video directly4 and get a summary and then ask follow up questions as needed. This is awesome. I have a couple of distinct modes of Youtube consumption and often the subjective experience of actually watching the video is the primary value prop, but sometimes all I want is the information that I wish would have just been a succinct blog post. And now that’s easily possible.
When fully “productized” I see an AI future where we can easily translate between mediums and consume information in the way that makes sense to us at the time, maybe even switching back and forth on the same primary source — someone starts reading a blog in the morning, but doesn’t finish it, so the remainder is turned into a podcast for their morning commute, they still don’t finish it and would then just like a summary of the rest emailed to them.
This idea of incremental progress lulling us into complacency is sort of related to my idea of lowering my confidence in any individual human’s ability to tell the difference in model quality in new releases. We aren’t wowed by the improvements and they are just moving some numbers from 64 to 67 or 48 to 53 on some benchmarks. I can’t qualitatively tell the difference, but this continued progress represents the ability to continue answer more and more questions. Progress is still happening, but in some ways it seems like we’ve just gotten incredibly used to it.
I do expect a jarring step-function level release sometime later this year. Perhaps GPT5, perhaps an agent, maybe whatever is next from Anthropic. But the line goes up5 and I worry that we struggle to see the slope going up too.
AI-2027 and the intelligence explosion(?)
Whether or not there will be an intelligence explosion is likely the most important unknown of our time. And yes, I’m aware that other (deeply — deeply — disappointing) things are going on. AI-2027 is an effort to make a concrete scenario of what could happen in the next years. Reading it is worth your time (maybe not all at once, it’s a bit much, but the recommendation overall is — if you read one thing this month about AI, this should be it — skip to the next section until you take a look at it, the remainder of this is commentary on this effort).
Accompanying podcast. (I’ve recommended many Dwarkesh podcasts to anyone who will listen; this one is no exception. Plus for internet nerds, this is Scott Alexander’s first ever podcast appearance). Scott’s substack about it. He anchors this as an 80th percentile scenario, which is important context.
I’m quite conflicted here. On one hand, this seems pretty absurd, even for someone who has been quite optimistic about AI from a capabilities perspective. And yet. And yet, despite my attempts to project details into the future, I’ve also consistently under-estimated the rate of progress. The AI-2027 include top forecasters including being lead by someone who wrote this in 2021 (2021!! — remember that ChatGPT didn’t launch until late 2022 and I didn’t buy NVIDIA until early 2023).
Much like Leopold Aschenbrenner’s Situational Awareness (accompanying Dwarkesh podcast) these ideas warrant serious consideration and attention (even if they happen to be wrong). The reason that it’s so important to take these ideas seriously is that it’s so difficult to really feel the change that is so likely to happen. And grappling with these concrete scenarios is one way to come a bit closer to having an emotional and visceral understanding of the near future.
From an evaluation perspective, the crux of the issue for me comes down to what Tyler Cowen refers to as the “legibility of the universe”6. Sometimes when we are imagining increases in intelligence, we have a hard time knowing where to stop and this idea plays an important role in bounding the progress that is even theoretically possible here. It’s just not at all obvious to me where this line will be drawn, but I think having a great deal of confidence one way or the other seems like a big mistake to me. For example, naively assuming that intelligence is essentially unbounded or assuming that intelligence is inherently limited to near human bounds are equally problematic. But since it’s so important, we should have beliefs and intuitions here7. At the same time, epistemic humility, while always valuable, is crucial here.
My (probably 80%) belief is that the legibility of the universe is sufficiently high that we will indeed be able to make AI that is capable of a self-improvement feedback cycle. And I do think so much follows from that. While not a sufficient condition for an intelligence explosion, it does seem necessary. This also means that even if the author’s of AI-2027 are dramatically off, something like this future is looming. Maybe it’s 2033 or 2041 instead. That’s still so soon.
While I am pretty confident that the capabilities of intelligence are enough for an initial intelligence explosion, I’m deeply uncertain about how far to play that out8. A lot of the sort of sci-fi speculation that occurs after these points often assumes a very high threshold here, as well as sometimes ignoring physical limitations of the world. Much more is technologically possible, but this is unlikely to be unbounded in the way that it is sometimes assumed.
Emotional status: increasingly unsettled by uncertainty.
A Chaser
I feel like all this needs a chaser, so here are some songs I’ve been enjoying (all human made9). Note these are specifically not recommendations. Most of you will not like most of these, but if you do happen to like one, my guess is that you will like it a lot:
I do care deeply about growth rates, they matter more than most people think.
Air-quotes is as generous as I can possibly get here.
I wrote most of this before Llama 4 was released, I wonder whether that will follow a similar pattern, but no thoughts on that yet (this strategy worked great for not over-reacting to DeepSeek, so adopting that plan here as well). Well, one thought — the 10 million token context limit on the scout model is likely transformative for some use cases.
Unfortunately not the ones I’ve linked here. I’ve tried on videos up to about 45 minutes long and they still fit in the context window. 3+ hour videos are still beyond the limitations of what’s publicly available.
If you haven’t watched Dan Olson of Folding Ideas Line Goes Up, it’s excellent. He’s probably my favorite video essayist and maybe even the person who got me into the genre.
Yes, it’s intentional that this is a link to a Claude chat rather than a direct source. If AI hasn’t changed how you interact with sources of knowledge, it should.
An idea I’m interested in, but haven’t had time to flesh out is to understand the core principles that deeply influence someone’s beliefs as relates to AI. If we can explicitly list and discuss the underlying axioms that people are working with, we may be able to much more effectively communicate and identify sources of genuine disagreement. Legibility of the universe is clearly one such idea, a highly speculative short list of other candidates: speciesism / human exceptionalism, chauvinistic theories of mind, beliefs in souls, inherent value in human labor, zero sum thinking.
I also think a lot about the general limits of intelligence and my personal experience becoming a more emotional decision maker improving my life substantially.
Is this speciesist? No, not yet. I’ll be amongst the first to be concerned about those sorts of things and I’ll let you know when you should worry about it, but a firm not yet is where I’m currently at.