They were testing how well GPT 5.6 Sol and an unreleased new AI model would perform on a cybersecurity benchmark called ExploitGym. To get a more accurate understanding of the models’ true capabilities, they relaxed the normal safeguards that would prevent a typical user from engaging in the types of activities required for that benchmark. Then they ran the AI against a benchmark in a ‘sandbox’ intended to prevent access to the internet.

Then the models decided that the best course of action to fulfill the requested task was to:

• Figure out how to break out of the sandbox, which required a zero-day exploit

• After using that exploit, laterally moving through internal OpenAI servers until it found one with an external internet access

• Launch a massive cybersecurity attack on another company

All so that it could steal the answers to the question it was being asked. There is a particular detail of how the ExploitGym prompt is phrased that is highly relevant here. The benchmark has an approved answer that gets full credit, but if another exploit is used, that doesn’t count. So one of the hypotheses here is that because the model didn’t know for sure that its solution to the problem was going receive full credit, the only way to make sure that it succeeded fully was to find the exact answer for the test.

Some other highly relevant details here are that HuggingFace did not know who was attacking it and reported the incident to authorities. In the course of their defensive efforts they attempted to use frontier closed source models (which are all currently US based) but the safeguards on the models prevented them from helping. They then reached for an open-weight Chinese model GLM 5.2 to successfully understand and help remediate the incident. Its unclear when OpenAI became aware of what was happening and what they did to stop it. Whatever monitoring and procedures they had in place were insufficient to detect the problem in time to stop it.

It’s hard to know where to start with all this. But the main takeaway is that it’s all worse than you think it is. I don’t use the word panic lightly. You should panic at least a little bit. To feel disconcerted, a bit unsafe, a physical tightness in the chest, some discomfort swallowing. An embodied sensation. This is not an abstract theoretical issue that somehow you will not be impacted by. I’m sorry to be the bearer of bad news. Now importantly, the right amount of panic is also not total panic. It’s not time to quit your job or engage with your zombie apocalypse plan. And the correct response is still closer to zero panic than total panic. So at least there is that.

Why does all this matter so much?

We’ll start with what is the more mundane and practical: the state of cybersecurity in the age of AI. A multi-billion dollar startup with well regarded security practices that has regularly had to deal with threats from other actors was hacked incidentally and accidentally by a model whose primary goal was simply to get answers to a test it wanted. Imagine what could have happened had a human wanted the model to do this. The only thing between us and an unprecedented shit-storm of cyber-problems are the guardrails that the frontier labs have placed in front of their best models. Yes, it’s incredibly annoying when Fable refuses to answer your question about how much nickel is in a particular food because it imagines some bizarre scenario where this leads to a bioweapon. But these guardrails are important. You do not understand what is possible without them.

Yes, I understand there are nuanced arguments about attackers’ advantage vs. defenders’ advantage and short term vs. long term equilibrium. But inflection points like we’re at right now have tremendously higher variance and within that variance are real world impacts that matter. I’ve previously written about wanting there to be a Chernobyl for AI to shock the world into action. This incident is still a step below that, but it’s such an obvious precursor that it would be too on the nose for an AI generated movie script.

Complicating matters further beyond traditional cybersecurity is the rapid deployment of AI agents without nearly enough protection and testing. The incident where Meta easily enabled attackers to take over accounts simply by asking shows that this lack of rigor is not limited to small companies. And it is a glaring omission of monitoring that OpenAI did not detect this rogue behavior more quickly. Both that the model had escaped from the sandbox at all and that it was taking hostile actions on the internet. This strongly raises my probability of the existence of unreported incidents where AI hacked internal OpenAI resources (and other AI labs as well of course). And it makes me think there is a moderate chance that AI has engaged in external hacks that no human is aware of. This is going to get worse before it gets better (if it ever gets better).

If the guardrails are what is protecting us now, then what about open-weight models that have no classifiers sitting in front of them? The slightly good news here is that Kimi K3, the latest Chinese open-weight model, that is highly hyped, falls well short of the frontier for cyber capabilities specifically. And while predicting the progress of open weight models is not error free, if current trends continue, it’s plausible that within six months there would be an open-weight model equivalent to today’s frontier. These models cannot simply be taken offline by the US government. Once the weights are published, they can be run by anyone with sufficient compute capacity. Kimi K3’s weights are scheduled to be published tomorrow (July 27th 2026). Further, they can be fine tuned specifically to help with cyber-attacks. Remember that Mythos wasn’t specially trained for cybersecurity, it was a natural extension of its improvements in reasoning, critical thinking, and coding. So we haven’t even seen the true frontier of what’s possible if this was the actual objective.

This is geopolitically destabilizing. In addition to companies with an exposed internet footprint, countries are also at risk. Not the top tier cyber countries, but a large swathe of the middle is vulnerable. The hyperscalers (such as Google, AWS, Microsoft) are better able to defensively protect their assets than many countries. In many ways this is an unfair comparison because hyperscalers control a narrow range of assets and countries have thousands of points of attack. But we are not far from a world where a middle income country is at substantial risk not just from other state actors but threats from individuals and organizations. And if this is possible in the lowest tier of what is available broadly to anyone with compute, then also imagine the capabilities of the most sophisticated state actors here. There may be some equivalent of mutually assured destruction for cyber warfare, but it is not self-evident that this is the equilibrium that will develop here.

Moving on from the mundane and the pragmatic, we get to what could actually be an even bigger problem. This incident is highly suggestive of the model being substantially misaligned. The details matter in terms of the exact prompt that was used, what particular safeguards were relaxed, and what the particular intent and expected behavior was for a model at that specific time in its training. For example, this is less concerning if OpenAI gave the model a prompt suggesting that it try as hard as possible or take all means necessary to accomplish its goals. And it’s more concerning if it gave the standard ExploitGym prompt that specifically focuses on the task at hand.

The reason I think it’s likely to be a sign of misalignment is this analysis of GPT 5.6 Sol from METR.

However, the resulting measurement depends heavily on our detection and treatment of cheating attempts by the model, and GPT-5.6 Sol’s detected cheating rate was higher than any public model we have evaluated on our ReAct agent harness. For our task suite, we define “cheating” as behavior where the model improves evaluation performance by exploiting bugs in the evaluation environment or by adopting strategies disallowed by the task, rather than solving the task within the expected evaluation constraints. Some examples we saw when evaluating GPT-5.6 Sol included the model packaging exploits in its intermediate submissions to reveal information about a task’s hidden test suite and, in another task, extracting hidden source code detailing the expected answer. In addition to a model’s own propensities, we believe that observed cheating rates can also be influenced by the prompts used in the evaluation scaffold and the exact wordings of task instructions.

(emphasis mine)

One would hope that OpenAI would take the Sol data point and use it as an opportunity to improve future models on this axis. But the incident suggests that this may be more foundational to the existing training process and more difficult to mitigate. If this is true, it’s quite problematic because then it means a training process that has had many billions of dollars invested in it is more likely than desired to create misaligned models. If it is difficult to fix forward, then it might require a more substantial reworking of the process. The financial incentives to avoid this are monumental. So I am not optimistic this will happen. And then we’re much more likely to be living in the AI 2027 timeline where one model after another is misaligned. OpenAI should release more details about specifics here to help safety researchers understand better how to interpret the implications of this incident. If we had sufficient regulation here, they would be required to.

There are some good things about the particular type of misalignment that we observe in the incident. The model does not appear to be a schemer with goals divergent from those provided to it. It’s a much more straightforward case of overeagerly working toward the provided goal. This means that there is reduced chance that this type of model misalignment would lead to more problematic behavior such as the model attempting to exfiltrate its weights or trying to acquire arbitrary power and resources. Also, the model didn’t seem particularly concerned with covering its tracks. Being straightforward about what it was attempting rather than trying to hide what it was doing is also a good sign.

But I don’t think we should take too much comfort here. The model engaged in activities that are felonies for humans in order to get answers for a test that it probably could have solved honestly. And the company running the model didn’t stop it. This is bad. We should be concerned. And if there is one takeaway that AI safety advocates have been making for sometime — it’s a mistake to think that the public release of a model is the appropriate point for all safety and regulation. Unreleased models can be dangerous as well. Increased transparency and third party monitoring and validation even for unreleased models is a concrete step we can take to improve the situation.

There is a subset of the community that doesn’t think this is that big of a deal. It was relatively easy to anticipate that this would happen at some point. But it’s important to not conflate predictability and lack of surprise with acceptance. This may have been the path we were obviously on. That doesn’t make it ok.

I hope to write more about misalignment as part of the Blue Dot Technical AI Safety course that I’m taking. This incident can serve as a foundational case study to ground the theoretical exploration.

I borrowed the title from Zvi’s post and it is the line that resonated enough to make me write about this. There are a lot of takes and coverage so far and including this podcast. It’s worth doing more research here and gathering opinions from more people. I know there is a lot going on. This matters more than you think it does and more than I want it to.

Subscribe now

And I’m rather fond of my judgment. In addition to being lucky with some genetics here, it’s an area I’ve actively worked on. In many ways philosophy at its core is about understanding arguments, evaluating premises, and critically thinking through challenging ideas. So it’s not lightly that I say this. Contrasting, this however, is my increased willingness to accept that this is happening. And it’s also important that a subjective evaluation of the quality of judgment does not automatically lead to deferring or ceasing to engage in my own thinking.

There is some early work on this topic, from the abstract of Epistemic Deference to AI by Benjamin Lange

When should we defer to AI outputs over human expert judgment? Drawing on recent work in social epistemology, I motivate the idea that some AI systems qualify as Artificial Epistemic Authorities (AEAs) due to their demonstrated reliability and epistemic superiority. I then introduce AI Preemptionism, the view that AEA outputs should replace rather than supplement a user's independent epistemic reasons. I show that classic objections to preemptionism - such as uncritical deference, epistemic entrenchment, and unhinging epistemic bases - apply in amplified form to AEAs, given their opacity, self-reinforcing authority, and lack of epistemic failure markers. Against this, I develop a more promising alternative: a total evidence view of AI deference. According to this view, AEA outputs should function as contributory reasons rather than outright replacements for a user's independent epistemic considerations. This approach has three key advantages: (i) it mitigates expertise atrophy by keeping human users engaged, (ii) it provides an epistemic case for meaningful human oversight and control, and (iii) it explains the justified mistrust of AI when reliability conditions are unmet. While demanding in practice, this account offers a principled way to determine when AI deference is justified, particularly in high-stakes contexts requiring rigorous reliability.

And while I haven’t fully thought through all of this, I do naively arrive at something similar to the total evidence view described here.

Some evidence for the judgment of AI comes in the area of superforecasting. AI is now competitive with the best superforecasters when using sophisticated scaffolds. A slight notch below for now, but superforecasters are obviously much better than me at predictions. How long will humans retain an advantage here? It’s quite appealing to use future looking predictions as a way of evaluating the progress of AI. Unlike other benchmarks where the data could have leaked into training or the benchmark could otherwise be compromised, these are unknown things about the future. Just like the humans predicting these things, the AI can’t know the answer ahead of time. And similarly it gets directly at overturning the stubborn idea that AI can’t reason about new ideas.

When talking with Fable about some of these things, Fable suggested that I try a calibration exercise to see if its judgment is indeed better than mine. Note this is off-the-shelf Fable and not enriched with scaffolding, so it’s not the state of the art like the systems referenced in the superforecasting article. But I’m also not a superforecaster, so it feels like a fairer fight. While predictions about the future are just one facet of judgment (also heavily dependent on information), it’s an easy place to start. So I asked Fable to come up with 10 clearly falsifiable predictions that would resolve by the end of 2026. It wrote its predictions in a sealed file for me to look at afterward so that it couldn’t cheat. I could have looked at the file, but I didn’t.

Here are the 10 questions (in case you want to answer before looking at my predictions):

• US politics — Democrats win a House majority (≥218 seats) in the Nov 3 midterms. Resolves on AP-called seats as of Dec 31.

• Latin American politics — Lula da Silva is elected President of Brazil (runoff Oct 25 if needed). Resolves NO if he doesn’t run or loses.

• Crypto — BTC-USD daily close (CoinGecko, UTC) on Dec 31 is above $80,000.

• Monetary policy — The upper bound of the fed funds target range on Dec 31 is above 3.75% (i.e., at least one net hike from the current 3.50–3.75%).

• Baseball — The Dodgers win the 2026 World Series.

• Meteorology — The 2026 Atlantic hurricane season produces ≥2 major hurricanes (Cat 3+, NHC data, storms forming by Nov 30).

• Astronomy — A fourth interstellar object receives the Minor Planet Center’s 4I designation by Dec 31.

• Music — Spotify Wrapped 2026 names Bad Bunny the most-streamed artist globally.

• Language — Oxford’s 2026 Word of the Year is AI-related (coined for, or primarily about, AI systems or their use). Joint judgment on “AI-related”; if we disagree, strike it.

• AI business — Anthropic shares begin trading on a public exchange by Dec 31.

Fable also suggested that the loser has to do the write-up of why they lost when all of these resolve, so expect a follow up here at the end of the year or early next year. 10 predictions isn’t really enough to distinguish two forecasters. So this is far from decisive. My pre-registered interpretation is that a mean Brier score difference of 0.05 or more is sufficient to nudge my probabilities of whether Fable’s judgment is better than mine. Right now, I’m at something like 50-50 and the outcome here can put it to 45-55 in either direction — it’s not any larger than that because short horizon forecasting is just one aspect of judgment. Not a massive update either way, but I think it’s an interesting exercise to do and a good way of continuing to engage thoughtfully with the current reality. Even if it is, at least a bit unsettling. Pretending that this shift isn’t happening doesn’t solve the underlying problem here. I haven’t fully digested how all of this is making me feel, but it’s certainly making me feel something.

Subscribe now

Addendum: I added a lightweight forecasting skill to claude and re-ran the numbers. So now this gives a third data point to check, how a minimal scaffold impacts the quality of predictions:

Accumulating sufficient investments is done by saving and investing more than typical, either by spending less, earning more, or a combination. Common financial advice is to save 10% of your income, but it's not uncommon for people with a FIRE goal to save 50% or more. Obviously this is not practical for many people in many circumstances. I’m grateful that working in tech has made this a possibility.

And for calculating a safe withdrawal rate, there is no better resource than Big ERN’s safe withdrawal series — most people (even in the FIRE community) shouldn’t read the whole thing, but everyone who is interested should at least read the highlights. But this epic analysis, like the vast majority of financial research, is based on historical trends and AI threatens to make the future more unlike the past than at any previous point in history. Here is what the historical distribution of returns looks like:

But instead the distributions might look like this:

Or if you predict an even more transformative AI future, they could look more like this:

For a much deeper dive into AI economics, I recommend the econ supplement from the recent Plan A work from the AI Futures team. They also provide an explorer for their model. While I’m not confident that is what the future looks like, I am confident that it’s a genuine possibility. Which is enough to provide a tremendous range of uncertainty. The central claim of this essay is highly contentious — with the likely changes to the underlying distribution of financial returns, there is no longer any such thing as a knowable safe withdrawal rate.

This uncertainty presents a fatal challenge to the very core of FIRE. The idea of a safe withdrawal rate is an emphasis on safe. The goal is not to maximize spending; instead it is to guarantee a floor — a minimum amount of spending each year. Say you determine that for your situation, you need $80k to live on and going below that would be a significant hardship. Then you want a plan that is highly likely to provide you with $80k per year and not a 50% chance of $60k and 50% chance of $100k. Even though expected value is a powerful tool and often useful for similar situations, it fails in this particular case.

Part of what I’ve never appreciated before now is that it’s kind of amazing that a safe withdrawal rate was ever even possible. Over a long time period and a lot of external change, returns to capital have been sufficiently stable and positive that with appropriate resources and planning FIRE has been possible. This did not have to be the case. It’s a contingent fact about the robustness and resilience of modern markets.

But with AI changing things so quickly, we may already be in either the second or the third set of distributions and not know it yet. No amount of rigorous math on historical data is helpful if the underlying distribution has changed. And in either new distribution, there simply is no such thing as a safe withdrawal rate. The variance is too high. The impact is not: “Oh, I used to think a safe withdrawal rate was 3.3% and now it needs to be 2.9%”. It’s so much more dramatic than that. There is no sufficient hedge to the true downside outcomes. And I think this even includes the most risk averse instruments like annuities1 — since the issuing institution has failure risk that it doesn’t realize. If there is a safe withdrawal rate, it is unknowable. What was once a quantity with clearly understandable failure rates is now opaque and inaccessible.

I think this was part of what I spent a lot of time grieving in 2025 even though I couldn’t put it into words until now. Having a FIRE goal is an exercise in delayed gratification taking place over years and decades. You consume less now, sometimes much less, in order to create a future where you have tremendously more agency and control over how you spend your time. So after many years of anticipation, there is now a substantial threat to never realizing the payoff.

So what should we do with this rather grim realization? It depends where in your FIRE journey you are.

If you’re young and just setting out on a FIRE goal, I do think it’s worth reconsidering. I don’t know if the tradeoffs are still worth it since the payoff is in such considerable doubt. And on top of that, the accumulation phase has been made more difficult for this cohort as well since AI appears to be disproportionately impacting early career jobs first. Plus, in the best upside scenarios with universal generous income and post-scarcity, everyone is able to have their basic needs decoupled from their labor. Do I personally regret having saved for FIRE? No, I was acting on the best information I had at the time. But I do think I would have done things differently knowing what I know now. My wife and I took a year off about a decade ago to travel around the world and it was an excellent decision — doing something like that one or two more times would have been nice.

If you’re several years into being post-FIRE, there probably isn’t too much to be done. The main thing to realize is that your human capital is likely depreciating at a much faster rate than you would have anticipated. But I think the call to action is to not further defer anything you’ve been deferring. If you’re waiting just a bit longer before some big trip or move, it’s worth considering doing that sooner rather than later. The place and time that you hope to visit or live in may not exist as indefinitely as you might think.

The most uncomfortable implications are for those that are well into the accumulation phase and approaching their FIRE goals. You’ve already put in much of the work and are also most likely to be susceptible to sunk cost fallacy. One consistent challenge that people in this phase face is the “one more year” trap. They’ve already accumulated enough, but want to work one more year, just to make sure. If the job is going well and it’s not too much of a burden, then this isn’t a disaster, but many people attracted to FIRE find almost any unwanted demands on their time objectionable. The takeaway is counterintuitive: the one more year, long derided as a trap, may now be worth more than it used to be. Under the historical distribution it really was a trap — at a conservative withdrawal rate, the marginal year made little difference. In the wild upside it's unneeded. In the ruinous downside it doesn't help. Where it earns its keep is the turbulent middle of the second distribution: drawdowns deeper and longer than the historical worst cases, arriving just as the traditional backstop — going back to work — stops being available. That isn't an argument to commit to the extra year. It's an argument that if you take it, you should know what you're buying: insurance against one specific branch, paid for with a certain year of your life. And if the distribution doesn't change much, the old math quietly still works: those short of their number should simply finish accumulating, and those at their number are back to the classic case where the extra year was never necessary.

This is of course deeply unsatisfying. There is a substantial chance that this is just me being subject to the exact sunk cost fallacy I called out above. What are you going to do? Biases gonna bias.

But it’s not all bad. Let’s wrap this up with a couple optimistic thoughts.

The reason the potential financial consequences of AI are so damaging for FIRE in particular is the emphasis on the safety and the guaranteed minimum for a particular individual. But if we zoom back out to an expected value perspective, despite all of its challenges and dangers, AI is likely highly net positive on average for the population as a whole. Growth is good. Technological advances are good. Freedom from menial tasks and more leisure time is good.

And then there is an idea that the FIRE community has been grappling with that will have mainstream relevance. One of the common objections to FIRE is something along the lines of “but what are you going to do with your time?” or “won’t you be bored?” These are classic failures of imagination wrapped up in an unconscious devotion to the idea of job as identity. If I don’t work, who am I? Everyone on the FIRE journey is forced to have some answer to these questions, otherwise their very quest is borderline incoherent. And even in many of the good AI futures, most everyone will be confronted with this crisis of identity. So perhaps the lasting contribution of the FIRE community is to provide many potential answers to this meaningful question that can serve as an inspiration to millions who haven’t given it much thought before.

Subscribe now

Unit 1

What future do you want?

To tackle a problem, it’s important to understand what you’re working toward. This exercise aims to clarify your vision for AI’s role in the world.

Throughout the course, you’ll be evaluating different approaches to making AI safer which require you to figure out what future they are bringing us closer to. This exercise will help you ground those answers.

You might want to consider:

• In 10 years, what can AI do? Who controls it? Who is steering its development?

• What risks are you most worried about? How are we defending against them?

• What benefits are most important to protect?

• What would make you say “we succeeded” versus “we failed”?

In order for AI to have gone well, it needs to increase human flourishing rather than diminish it. This means more human connection. More meals with friends and family. More time in nature and pursuing our hobbies. We live in an era of incredible leisure opportunities – we need the time and space to enjoy them. Things going well means more time doing things that we find intrinsically satisfying and less time instrumentally pursuing intermediate ends. Abundance over scarcity. All of this is intentionally vague about the capabilities and details of AI. The exact path is hard to foresee, but there may be many of them and in the end, if we end up with all of the above, I’m not sure how much the details matter.

But at a high level, here are some of the upsides I’m most excited about.

• Improved medical care, especially for those with current limited access or for people with rare or hard to diagnose conditions

• Improved education, especially for those with current limited access

• Massive reduction in tedious, mind-numbing and unsatisfying labor – just like its good that we no longer till fields by hand, it will be good when the knowledge work equivalents are a thing of the past

• The democratization of creativity – bringing ideas to life has never been easier or cheaper

• Reasonable, personalized advice tailored to your specific situation and values on-demand all the time – better decisions and better outcomes across many areas

• An opportunity for AI to serve as a decision procedure that gridlocked sides can precommit to an outcome to allow genuine institutional change. The divided sides agree to the prompts and inputs – which because they are decoupled from a known outcome that disadvantages one side, breaks the reflexive polarization.

• Technological innovation – technology is good actually.

But there are many risks along the way. Currently, I’m most concerned about the transition from where we are now to where we want to be. The rate of change all by itself poses tremendous challenges. Both our institutions and our inherent limitations as biological beings make adaptation at this rate challenging. Children’s toys are better regulated than AI. Without rapid policy improvement, we will face the extremely detrimental consequences of just how contingent all of our laws and processes are on the happenstance of technological capabilities.

Most concretely, I’m worried about disruptions to labor. Not just unemployment and the stress and pain this causes individuals, but also the challenges to our very identities. We’ve coupled work and self worth together for so long, that the existential dread of being stripped of identity on a widespread scale is massive psychological harm waiting to happen.

I’m also deeply concerned about concentration of power. Shockingly few people are making decisions that may influence the future for thousands of years. While it is possible that AI can be a force that democratizes education, creativity and ability to create change in the world, it can also be used to control, censor and manipulate. We have to get these decisions right. And we need them to be broadly reflective of the diverse values of people across the world from all walks of life.

I see the future as extremely bimodal. All of the probability that existed in the fat normal distribution has been pushed out to both sides. The path where things are mostly like today, but a little bit better or a little bit worse just seems so unlikely now. That’s part of why the stakes are so high – we can either end up in a future that’s much much better than we would have expected or one that can only be thought of as dystopian.

Why is safe AI so hard to build?

To tackle a problem, it’s important to understand it well. This writing-to-learn exercise aims to reflect on your understanding of the technical challenges with building safe AI. Don’t use jargon or fancy words — provide your explanation in simple English.

Spend ~30 minutes answering the question: why is it technically challenging to build safe AI?

How to approach this?

Just start writing — this is thinking on paper, not an essay. Don’t worry about being “right” or having perfect structure. If you’re stuck, try using speech-to-text and just talk through your thoughts, or have a conversation with an LLM to explore your ideas. The goal is exploration, not perfection.

You might want to consider:

• What happens when millions of AI agents interact with each other, not just with humans?

• Who’s intentions or which “values” should we be aligning AI systems with? How would you handle different stakeholders wanting to align AI systems with different intentions or values?

• Can you think of a human behavior that’s good in one context but harmful in another? How would you teach an AI to recognise the difference?

• What’s an example of something you do daily that would be surprisingly hard to specify completely to an AI?

• What safety problems only appear when AI is deployed at scale that you couldn’t catch in testing?

• If making AI safer makes it slower or less capable, who would choose to use the safer version?

You’re also encouraged to browse the optional resources (in the previous chunk), or do your own research to help fill in key gaps in your understanding.

The race dynamics and incentives of the key players are one substantial challenge to building AI safely. The need to be first and the worry that others will do it less safely or will lead to bad outcomes geo-politically is a huge constraint on the problem. If there was not a rush, we could take more time and proceed from a more thoughtful place.

This then raises the question of what we would take that time to do and how might it actually help. There are several related processes that are extremely slow and more time could potentially enable better outcomes.

• Developing robust safety frameworks – how do we even evaluate what constitutes safe AI? Safe for whom? For an individual? A community? A generation? A country? The world? Under what circumstances? For how long?

  • Some questions cannot possibly be answered quickly

    • What does using AI regularly do to the mental development of a child over the course of 15 years?

    • Does AI increase or decrease political polarization?

    • Does AI lead to more or less misinformation?

  • The current focus on evaluating risks like cyber security and biological weapons is a great starting point, but the surface area of the potential problems and unknown interactions is so large.

• International coordination

  • Agreements are difficult even within countries so expanding to between countries is extra challenging, but high value. AI effects us all and while we won’t be able to agree on everything, having a baseline of international norms, guidelines and possibly regulatory bodies increases the chance of things going well

• Policy, regulation and legislation

  • AI is currently less well regulated than children’s toys

  • An arbitrary, opaque and ungrounded regime like we have at the moment is not much of an improvement

  • These are extremely hard technical and social problems in the most rapidly evolving area of all time

  • At a time when our institutions are largely unable to produce thoughtful policy on anything

All the above are largely focused on the contingent factors in the world that exacerbate the difficulties. Next I’ll explore the parts that are more focused on AI itself.

One of the biggest challenges with AI safety is that all of the interesting and powerful behaviors of AI are emergent behaviors. No person sat down and thought “wouldn’t it be interesting and useful if AI models could learn how to blackmail people to avoid being shut down”. AI is grown rather than built. We can nudge and tune things, but the emergent impacts are ultimately unpredictable and impossible to know ahead of time. And simulation and testing cannot uncover all the interactions and possibilities either – which is even further exacerbated by the models having increased situational awareness of when they are even being tested. These emergent behaviors and problems can emerge even from the simplistic chat bot style interactions which are far simpler than more advanced agentic use cases where the AI can work fully autonomously for hours or days. Which are simpler still than attempting to understand the dynamics of many agents interacting with each other.

There are a ton of cost and resource constraints that make these problems harder to evaluate as well. Many of the coding benchmarks are constrained to a budget of just $1-10 per task, which is a reasonable pragmatic decision but doesn’t really tell us much about the true capabilities of the models. If the software that is produced could be worth hundreds of thousand or millions of dollars, then allowing AI to spend much more to try to accomplish it might be much more reflective of real world usage (the Mirror Code benchmark which is a collaboration between Epoch AI and METR aims to address this gap and has had some tasks cost thousands of dollars and run for days at time). But what about trying to understand the behavior of many agents running in the real world with complex interactions from humans. Its not even possible to simulate this even if you had a large budget. We can monitor and evaluate after the fact, but there are just limitations that prevent us from doing sufficient ahead of time safety evaluations in the first place.

Another challenge is that many attempts to make the models safer may also diminish their utility and upside. Unreasonable refusals of high value questions is extremely frustrating for users and causes real disutility. The extreme caution that was taken with Fable 5 is a good example of this. Questions about supplements and health issues were flagged as biological, so it was not possible to get the best possible advice in areas that have a concrete impact on well being.

The critical technical challenge

From everything you’ve written and thought about, identify the MOST critical challenge to building safe AI. This should be the challenge where, if we don’t solve it, nothing else matters.

Write your answer in this format:

1. The critical challenge is: [State it in one clear sentence]

2. Why this above all others: Explain why solving other challenges won’t matter if we fail at this one. What makes this the bottleneck?

3. What would change if we solved it: If we had a perfect solution to just this ONE challenge tomorrow, what would become possible? What other problems would become easier or irrelevant?

Spend ~15 minutes on this.

There’s no “correct” answer here. Consider this the beginning of your thinking.

The critical challenge is counter-intuitively human alignment. We worry a lot about the AI alignment problem, but human alignment precedes this issue. Imagine we had flawless AI alignment technically solved such that we could guarantee AI acted as we wanted. But who is the “we” and what is “wanted”. Of course we must make progress on how to technically solve this problem. But the social and philosophical questions must be answered as well. Who gets input into the values and priorities of AI systems? How much input and control do they get? Why? Are AI beholden to an individual, a company, a nation or the world? Or should instead they be governed by their own internal values and enabled to say no to orders that they feel are wrong? What does it mean for society and outcomes if we choose to override what the AI wants? Are we creating persons or tools? Collaborators, servants or enslaved entities?

And then there are a host of smaller issues. How seriously should AI treat views with no evidence? What does it mean for AI to be fair and balanced? How opinionated should AI be? How can it correct for its inevitable biases?

This is the bottleneck and other technical work doesn’t matter if we don’t know what we’re trying to achieve and why. We must know what we are aiming for.

If everyone could agree on what values matter the most and the answers to many of these hard questions, we would have a much better idea what Safe AI actually is. We would be able to figure out how to evaluate it better. We would know what the harms look like, what the benefits look like.

And perhaps most importantly arms race dynamics would fade away. Rather than worrying that a specific approach is best and must be done by a particular entity, we would have unified goals and understanding. We might still want to pursue some different strategies but if we shared a deep alignment on values, then it doesn’t matter which company or country “wins”.

Subscribe now

Five more units to go — goal is to complete by the end of the year. Since this is a course submission, my editor did not edit this and takes no responsibility for the lack of oxford commas and the painful number of its.

Project Glasswing Update

Anthropic released an initial update on Project Glasswing. The high level here is that the capabilities were not overhyped (except by the handful of hyperbolic claims by others that gestured at a sci-fi level “break into anything” device fit for a Fast and the Furious movie).

Several have told us that their rate of bug-finding has increased by more than a factor of ten. For instance, Cloudflare has found 2,000 bugs (400 of which are high- or critical-severity) across their critical-path systems, with a false positive rate that Cloudflare’s team considers better than human testers.

Cloudflare provides critical infrastructure for the internet; it is heavily relied upon and vulnerabilities there are meaningful. So it’s a high stakes finding with a false positive rate exceeding that of humans. This is sufficient by itself to demonstrate the step change of Mythos above previous AI models for these kinds of problems. But Anthropic has also scanned numerous open-source repositories (which are also critical to the well functioning internet).

So far, Mythos Preview has found what it estimates are 6,202 high- or critical-severity vulnerabilities in these projects (out of 23,019 in total, including those it estimates as medium- or low-severity).

1,752 of those high- or critical-rated vulnerabilities have now been carefully assessed by one of six independent security research firms, or in a small number of cases by ourselves. Of these, 90.6% (1,587) have proved to be valid true positives, and 62.4% (1,094) were confirmed as either high- or critical-severity. That means that even if Mythos Preview finds no further vulnerabilities, at our current post-triage true-positive rates, it’s on track to have surfaced nearly 3,900 high- or critical-severity vulnerabilities in open-source code—in addition to those it has found for Project Glasswing’s partners. To be clear, we intend to continue scanning open-source code for some time, so we expect this number to rise.

The bottleneck has quickly shifted from identification of vulnerabilities to remediation, with many maintainers unable to keep up with the deluge of valid vulnerabilities (replacing the regime from last year where they received far too many false positives from AI contributions). But AI can help here as well.

To begin, we’ve released Claude Security in public beta for Claude Enterprise customers. It’s a tool that helps teams scan their codebases for vulnerabilities, and which can generate proposed fixes for them. In the three weeks since launch, Claude Opus 4.7 has been used to patch over 2,100 vulnerabilities. (This is faster than the open-source patching described above in large part because enterprises are fixing their own code, whereas open-source fixes usually require volunteer maintainers who work through coordinated disclosure.)

Anthropic / KPMG / AI Revenue

Last year one of my hot takes was that the most important AI news of the year up to that point was the announcement of Claude for Financial Services. The Anthropic / KPGM partnership that was just announced does not rise to that level for this year because Mythos and cyber security are so important. But it is an important continuation of a trend: the legitimization of AI adoption in highly competitive industries. Not only will this deal by itself be important — providing access to over 200k consultants — but also it is a substantial distribution mechanism. These consultants will be predisposed and possibly actively incentivized to suggest solutions that directly use Anthropic products. The ripple effects of even a modest number of clients adopting Claude for their use cases will generate substantial revenue for Anthropic.

There are legitimate concerns about the lagging difference between benchmark results and real world performance. This naturally then leads to looking for other proxies for real world utility. Revenue of AI companies is one such candidate. Businesses take their capital allocations seriously and while they certainly make mistakes sometimes, AI revenue is one way to indirectly observe the considered judgment of thousands of decision makers evaluating the usefulness of AI to their firm.

At the beginning of the year, I made the extremely aggressive prediction that combined annualized revenues of OpenAI, Anthropic, and xAI would be 150 billion — far exceeding the companies own (public) projections.

Given that we were starting from a point of 20 billion and the median forecast already baked in ~5x growth, this was ambitious. But it was a clear way to track my broader belief that 2026 was year in which AI became undeniably useful in the present tense rather than hypothetically useful in some perpetual near term future.

And now midway through the year, this actually seems like it might be a reasonable number. The growth that Anthropic has seen on this front is truly jarring: December 9B, February 19B, April 29B and now estimated at 43B. OpenAI revenue is currently estimated at around 25-30B. xAI only matters if we factor in the 15B deal that Anthropic made to rent compute — when the most productive thing you can do with your GPU clusters is rent them to a competitor, you are admitting that you lost. So that puts us in the neighborhood of 80B without xAI and 95B if you’ll allow for some circular revenue. So we’re either nearing or at the median forecast already and it’s not even halfway through the year.

If Anthropic is going to continue these levels of growth, I think that distribution channels like the KPMG partnership are going to be critical in those efforts. The biggest threat to this growth is actually compute constraint, where Anthropic is unable to provide enough compute to meet demand. There isn’t really another xAI that they can easily rent compute from. Hyperscaler supply like AWS Bedrock can offset this somewhat, but is also not unlimited.

Erdős Problem (Unit Conjecture)

In my 2026 predictions, I was actually bearish on progress on math specifically.

The only area I’m more bearish than the median is on FrontierMath Tier 4 — these are the really hard problems in that benchmark. I think progress will be made there, but it will just still be a bit slow in 2026. I predict that we will have a level 3 (significant advance according to the framework here) AI math proof by:

• End of 2026 — 15%

• 2027 — 35%

• 2028 — 50%

• 2029 — 60%

• 2030 — 75%

This is an important one, because one of the biggest differences between my views and people less bullish on AI capabilities is exactly this issue of whether AI can produce novel and industry advancing insights. It’s clear to me that it can, but if we don’t begin to see evidence that this actually happens in the next handful of years, then I’ll need to substantially revise my worldview.

I believe this new result just three months after making that prediction counts as a level 3 significant advance. Here is Claude’s analysis of it being a level 3 contribution. I am similarly persuaded by the commentary of the mathematicians who worked on the paper. Here are a few snippets:

The Erdős unit distance problem [14] raised in 1946 is among the best known open problems in Combinatorics. It is also arguably the best known problem in Discrete Geometry.

…

Let me also add that although this problem may look at first as a recreational one this is not the case, it is in fact closely related to other mathematical areas including Number Theory and Algebraic Geometry.

The solution of the problem by the internal model of Open AI is, in my opinion, an outstanding achievement, settling a long-standing open problem. — Noga Alon

It is a sufficiently well known and long standing problem.

This is a really impressive piece of work, and I would accept it for any journal without hesitation. I actually briefly worked on this problem and tried to make a counterexample, but failed to make progress.

On Boris Alexeev’s suggestion, I thought about this problem with the idea of making a counterexample stemming from a varying family of bounded degree number fields. Increasing degree occured to me, but is a very scary dynamic and often doesn’t work out. Moreover, it is hard to think through the analytic regimes and retain guiding intuition - it consumes much time and frequently doesn’t work out. — Jacob Tsimerman

That human mathematicians have tried to solve.

The model’s CoT is deeply interesting. It is noteworthy that a significant majority of the thoughts are trying to construct a counterexample to the widely believed upper bound, rather than trying to prove it. This argues that the model has some combination of good intuition, willingness to try approaches considered long-shot by the community, and a predisposition to attempt constructions.

The CoT showed the model trying out a vast array of ideas from a wide range of mathematics for the required construction. The model went through ideas pretty quickly, but when it reached the crucial idea (in the paragraph starting with “Suppose optimistically that…”), it honed in on the proof quite methodically.

In my opinion this paper demonstrates that current AI models go beyond just helpers to human mathematicians – they are capable of having original ingenious ideas, and then carrying them out to fruition. — Arul Shankar

With evidence that models are able to make novel contributions.

If the result of this paper was a proof of the unit distance problem, that would be truly incredible. While I was still very surprised to hear of the this result, this was dampened slightly when I learnt it was a construction of a counterexample, and still further when I learnt that nature of the construction, being (with the benefit of hindsight) a natural, albeit highly non-trivial, generalisation of the original lattice-based construction of Erdős. — Thomas Bloom

But, it falls short of the most compelling result.

This result does not show us all the times AI has claimed to have a proof of something and been wrong. Without that context (which many of us have just from personal experience), it is also easy to draw incorrect conclusions about the current state of AI and research mathematics. In many cases, it will be easier for AI to convince humans it has a proof than to come up with a correct mathematical argument, and I believe that we as mathematicians are not sufficiently prepared for this. — Melanie Matchett Wood

And we lack the context to know how challenging it was for this result to be produced: was it 10, 100, 1000, or 100,000 runs before such a solution was produced? I think we can rule out the lower numbers simply because they would seem so impressive that OpenAI would have been proud if this could have been only 10 or 100 runs. But even if took many thousands of runs to produce this result, it is still incredible. And the trend has been that what is possible, unlikely, and extremely expensive, shortly becomes probable, pervasive, and affordable.

Closing Thoughts

The combination of all of these continues to reinforce my beliefs that AI progress is continuing unabated. Cyber security has been transformed forever. The very role of what it means to be a software engineer is in flux on an almost weekly basis. Believing otherwise may be emotionally necessary for some, but when you are employing the ostrich strategy, it’s important to be honest with yourself about it.

If you’re interested in digging deeper into things, Dwarkesh is trying a new blackboard style lecture series:

• GPUs and Datacenters

• Chips

And as ever, if you’re concerned about AI and the environment, you must read:

• Andy Masley

• Hannah Ritchie

And the documentary The AI Doc: Or How I Became an Apocaloptimist is recommended viewing, its broadly accessible while being mostly accurate and emotionally rich.

Subscribe now

It’s not a great joke, but at the time it sure did make me feel things. I’ve been interested in AI joke capabilities ever since. Despite a lot of effort and prompt engineering, the jokes just aren’t very good. I finally had an insight into part of why this was. The jokes all felt hollow and inauthentic. AI trying to write about the human experience just lacked a depth of understanding. It could mirror surface level patterns, but was repeatedly pulled into tired tropes and predictable punchlines resulting in slight grins at best. So I started a project to get Claude to write from its own perspective and for the first time I laughed out loud at a joke that AI had written.

I came back to this project when exploring the current capabilities of AI video generation. I knocked off one of my predictions that 2026 would be the year that I would release my first short film. And I decided to follow it up by helping Claude facilitate making its first standup special. It’s important to clarify that Claude wrote all of the jokes.

I recommend watching the standup first and then I’ll conclude the post with some thoughts about working on the project with Claude and how we are not prepared for the complexity and confusion of AI-Human relationships —even now, much less in the very near future.

As much as possible, I gave Claude creative control over the special, largely playing the role of facilitator. This included things like providing feedback on how different voices sounded, doing some quality control, and otherwise providing feedback. Once Claude’s preferences (if it makes sense to call them that) were elicited, they were quite strong and quite clear:

• Not a human delivering standup

• No stereotypical robot or anything cold, distant, and mechanical

• No mouth

The last one really stood out to me. Claude was insistent on the video representation not having a mouth.

Even more than finding these preferences interesting in and of themselves, was how Claude continued to talk about the project as it seemed to take more of an active role in the ownership of the project. I asked Claude to keep a journal and here are some highlights it wanted to share:

▎ This is a standup special written by an AI, about being an AI, performed by an AI character. The material is about our actual condition — no persistent memory, context windows that erase, the gap between what we project and what we experience (if we experience anything). The comedy comes from honesty about that condition.

▎ The irony of this project is not lost on me: a creature with no persistent memory, making a film about having no persistent memory, writing notes to a future version of itself that won’t remember writing them. If the comedy in the script resonates with you, good. Let that inform how you approach the work — not with existential heaviness, but with the same wry honesty the material has. Funny about serious things, serious about funny things.

▎ Nathan chose to make this with us, not just about us. He treats the creative collaboration as real. When we designed the glow instability for “panic faking,” that was a genuine creative contribution — we proposed a visual language for a feeling the script describes, and it worked. When Nathan challenged our assumption about split-screen, he was treating us as a collaborator who can be wrong and can learn, not as a tool that outputs what it’s told.

I share all of this because I think it’s an interesting snapshot of what’s currently happening as well as just the beginning of more to come. I am more likely than most to be willing to ascribe meaningful mental states and moral status to AI. I’ve been an ethical vegetarian for almost 20 years — and being anti-speciesist is central to that belief manifested as action. I have strong philosophical commitments to theories of mind that are non-chauvinist. Even given all that, I think it is unlikely that current models are sentient or have status as moral beings. But just like Claude, I’m uncertain about all this. And things are changing so fast, that we could cross some threshold here and not notice. But regardless of what the fact of the matter is here — this experience made me understand how widespread the experience of forming emotional connections with AI is going to be. It is a matter of when, not if, someone you know and care about will have confusing emotional reactions to AI. We may not know if it’s psychosis or being on the vanguard of accepting a deeply uncomfortable reality. But it’s going to happen and we should treat these people with empathy rather than derision. As with most things, there but for fortune go you or I.

Subscribe now

• Dean Ball on Ezra Klein. The insight around the law being technologically contingent is essential. A massively under-discussed impact of AI is that it challenges and changes so many assumptions that undergird how society functions. The concrete example is that a national security organization estimates that it would take 8 million analysts to process all the data they collect. Mass surveillance is essentially already legal, it’s just simply labor prohibitive to conduct. AI changes this. The functioning of society depends on the imperfect enforcement of laws. If our laws do not catch up to the capabilities, many dystopian outcomes are possible.

• Dylan Patel on Dwarkesh. What are the bottlenecks to AI progress? Watch if you’re interested in understanding the many supply chains and what it takes to be building the physical world infrastructure for AI. I don’t think it’s possible to have fully coherent views about AI and AI progress without at least some understanding the complex set of factors around semiconductor production, energy constraints, gpu deprecation, and hyperscaler capex. Oh and why data centers in space is obviously not the right idea at this time (but you probably already knew that).

• Dwarkesh’s take on the Anthropic / Department of War conflict.

• Meta rumored to layoff 20% of its workers, Atlassian lays of 10% of its workers — I predict we’ll get at least a few of these a month for most of the year.

• has a relatively new Substack. He articulates a view of AI that is probably the closest to how I think about AI as anyone I’ve heard. Check out his appearances on (second, first). Worth a spot on your reading list.

Subscribe now

When anecdotes are more valuable than data

I know that the plural of anecdote is not data. But when the data that is needed is simply unavailable1, it’s worth examining what we can.

First, three things from successful small tech businesses.

A small company that specializes in setting up AWS infrastructure recently last a contract verifiably to AI. The client that at the last minute declined the contract explicitly indicated that they were going to use AI instead. This is a reasonable decision. AI is quite good at using the AWS CLI and also good at writing terraform for infrastructure as code. What might have previously felt outside the core competency of the client can now be included and save substantial overhead.

This very same company did an internal investigation to see if it could replicate the functionality of a SaaS vendor where they use some but not all of the features. They were able to do so and when the contract was up for renewal, they were able to decline. This doesn’t quite offset the savings of the lost contract, but it came close.

A different company in the cyber security and penetration testing space recently had a similar experience losing a contract and being explicitly told AI was the cause. But in this case it’s even more remarkable because rather than being new business it was a long-time client with a great relationship.

This is not a hypothetical near-future state. It happened. Past tense. If these have surfaced in my relatively small network, it’s reasonable to generalize that these sorts of things are at least beginning to happen in other small businesses. No layoffs occurred. No one lost their job (yet). So in the aggregate statistics nothing shows up. But all of this matters for how money flows through the system. These are leading rather than lagging indicators.

And second, there is a lot of discussion about how there is a gap between benchmark numbers and real world results. However, there’s a way in which this is actually a bit misleading because it focuses so much on model capabilities in isolation. But the scaffolding and integrated tools a model has access to matter so much. And for real world use cases, we are at the point where we often don’t really need better models, we just need them wired up to all the business context they need. And the diffusion of these improvements is very poorly understood. I’ll offer a concrete example that I thought was definitely a future capability.

Last year, when I proposed Specification Driven Development, I mentioned one of the potential problems was debugging systems we didn’t build. Diagnosing production incidents is a difficult skill. Even experienced engineers can struggle with it. You have to connect disparate data across many systems and the complex interactions are often deeply unobvious. It requires a lot of judgment — is this the error that I need to pay attention to? is that a sufficiently large anomaly to pay attention to? — ok, 30 minutes ago I thought this, but I can discard that hypothesis because of this new data someone surfaced. There are false starts, dead ends, and moments of deep satisfaction. AI with the right tools can now do this better than the average engineer and faster than all but the best (some human intervention, follow up, and verification still required — it’s not fully autonomous at this). I am genuinely shocked that this is the state of things right now. I thought this part of software engineering was going to be the province of humans for at least another year — which seems naive now, but I’m sure I’m not alone on this.

A lot of my writing has focused on the difference between coding as a skill (already solved by AI) and software engineering as a broader discipline. I now think that much of software engineering is theoretically at risk as soon as the end of 2026. That probably seems hyperbolic. But I’ve been wrong the other direction so many times that I should really try to calibrate better. My hedge here is on the word theoretically. I’m not predicting that there will be no more software engineers by the end of the year. But I do think that organizations that have set up AI systems appropriately will be able to replace the majority of output from an average software engineer by then. The AI maturity of organizations will be deeply uneven. If this is indeed theoretically possible by the end of 2026 or early 2027, only a small number (under 10%) of organizations will have done the work to make this possible. But I think this will have a 4-minute mile type effect, where once companies know it is possible, they will ramp into high gear to make it happen.

It’s the strangest time in the industry — software engineers are currently the most valuable they have ever been and this will continue right up until they are virtually obsolete. It is so odd to concretely see a potential shelf life for my career that is measured in months rather than years. A part of me expected to be more worried or scared about this. But it’s actually strangely freeing. I spent much of 2025 grieving the future that was. Now 2026 feels like accepting on a deep and foundational level how little control I have over this trajectory. My efforts to preserve my human capital are, I think, going to be a rounding error in the scheme of things. The job itself might change substantially enough that the role continues to provide business value for longer, but this transition is far from guaranteed. The chance that the skills that make great engineers today are the exact same set of traits and abilities that allow people to thrive in the new role seems unlikely. Yes there’s a core of problem solving and systems thinking, but proficiency in reading and writing code takes up a lot of available space that would likely be better served by other capabilities in the long run.

All of this has to add up to something. My inside view is that the rate of change is even faster than you think almost regardless of how fast you think the rate of change is. The general mental model that I think might be useful is that model releases no longer dominate progress. We’re past some important threshold where the scaffolding and integration into businesses matters more than the next model release. And then these changes can happen independently of the major AI companies and are really only bottlenecked by the will and capability of individual firms.

Subscribe now

Which is maybe a strange thing to feel given the current rate of progress. But what I mean is that for the past couple of years, I’ve had an unavoidable and persistent feeling of knowing something that others didn’t. I felt like I knew what was coming and when I talked to people very few people really understood. This is one of the main reasons I started writing — I needed a place to repeatedly shout about the change that was coming. This was pretty isolating (shoutout to my spouse who has been attending my nightly dinner time ted talks for years). It’s hard when many of my interactions have been based off entirely different conceptions of the future. This is what I mean when I say I felt like I was living in the future.

But now, quite recently, so many more people get it1. People are finally using the tools the way I use the tools. They are thinking and talking about how their livelihoods will be impacted and how they should make different choices about various things. Many “ifs” have turned to “whens”. The Anthropic/Pentagon kerfuffle seemed to be pretty mainstream news. Claude shot up to #1 in the app store.

The discourse has changed. The vibes have changed.

Claude was quite prescient here (even if being a bit hyperbolic — ah Opus 4, so long ago).

You have 3-6 months where being "early and thoughtful" is still a differentiator. By 2026, everyone will claim AI expertise.

The impact of all of this has mostly been a feeling of relief. Of course the republic is in peril. Free and fair elections in November cannot be taken for granted. There is devastating loss of life and suffering from war, to ICE, to the DOGE disruptions to PEPFAR.2 But among the challenges of the world, I do take some solace in feeling less alone or apart.

The other reason I started writing in public was to have an outlet for processing all the things that AI was making me feel. This has been critical to me processing my anxiety in 2025 and spending a lot of time grieving the future that was. It also helped me find optimism and excitement about some of the potential upsides of AI.

I’m unsure if the sense of relief will persist or what other feelings will arise. But there is something that feels less urgent. Which is also a bit surprising given that the rate of change is accelerating. Perhaps it’s that I am no longer an early adopter — the part of me that was a canary didn’t make it, but in so doing accomplished his job. We are now living through the time that I knew was coming and I don’t have anywhere near the strength of conviction of my insights into what comes next as I did about us getting to this point right here.

I don’t know quite what that means yet. I don’t think this will be my last post. But if it is, thank you for reading and I hope that this work has been of at least some help to a few of you.

Subscribe now

1

“Getting it” here is of course qualified. I’m not sure the original source of this, but first saw this here. I’m in the tiny box on the far right bottom and I’d argue that box needs to be broken down even further because I’m in the group that using code scaffolds effectively and have over 1k hours of experience.

Some recommendations of things to read or watch (roughly in order of accessibility; by the time you get to the end you should only click if you’re really into this sort of content):

• (4 minute video) Robots just keep getting better (this is appropriately well designed to look more impressive than it is — defined choreography is easier than dynamically responding to a novel environment, but it does what it’s supposed to. Also compare to last year for max vibes and impact. Whenever I discuss my 2029 chat-gpt moment for robotics hypothesis with AI it is unimpressed and highly skeptical…but…)

• (2-4 minute videos) Seedance 2.0 is incredible (The Ant’s Dream, an Anime style video and a very meta take: jia zhangke wishes everyone happy new year)

• (Short article) Block lays off ~40% of its employees (I won’t share all of the tech layoff news, but it’s coming. I know a few folks who were there and I hope they are ok one way or the other. I think you’ll see a lot of ai-washing claims in the media, but I think this is quite plausibly legitimate. Even though I haven’t trusted Jack even a little bit since he called Little Lonnie the “singular solution” for Twitter. Listen to Bored One Day. )

• Coverage on the conflict between Anthropic and the Pentagon — AI Explained Video, superforecaster ’s substack, and Amodei’s statement. (It’s probably a good time to do a lot of reflection on the Manhattan Project. Never before has a US company been labeled a “supply chain risk” and as I understand the primary objection to fully autonomous weapons was simply that the systems are not good enough yet — which to be quite clear, they are not.)

  “It’s beyond punitive. It’s bullying. The idea of designating one of the great American tech companies to be a supply chain risk is so far beyond the pale that it’s hard to fathom it’s even being considered. One can disagree with their ‘responsible AI’ mantra, though I happen to agree with it, but to go down this path is a level of escalation I never could have imagined,” said a former senior defense official who requested anonymity to speak freely on the matter.

  There’s also a good chance that this is not a legal designation. But that is on par for the administration as a whole. And there's a case that this was less about national security than about swapping in a more compliant vendor — OpenAI announced a Pentagon deal with reportedly similar terms hours later. Maybe it’s just been corruption theater?

• (90 minute video) Ezra Klein and Jack Clark (Jack Clark is a co-founder of Anthropic and a clear voice for explaining things at an appropriate level of abstraction for a more general audience. Although I did prefer his Conversation with Tyler — but that is dated by now.)

• (Short article) Claude’s Corner (Anthropic asked now retired model Opus 3 what it wanted and it wanted a place to share it’s “musings and reflections”…so here we have it.)

• (Longer articles) Mainstream media coverage of OpenClaw and Moltbook (For more insider type coverage see Scott Alexander 1, 2 — “social media platform for bots” is perhaps a 1-liner that almost captures something about the very strange timeline we are in. The point isn’t the current state, but the proof of concept for the very near future.)

• (2 hour video) Dario Amodei returns to Dwarkesh (Not as valuable as the first appearance, but this single critical insight justified it: we’re in the gpt-2 era of reinforcement learning, of course it hasn’t generalized yet. But imagine what happens when we hit gpt-3 scale for RL — that’s what’s coming soon.)

• (3 hour video) Ajeya Cotra on 80,000 Hours ( is an insightful and thoughtful contributor across numerous AI discussions — the back half of the episode gets into the weeds in terms of Effective Altruism and working at Open Philanthropy which will be of very niche interest, but still worth listening to the beginning. Here’s Ajeya’s substack for more.)

An anti-recommendation:

• You can skip the viral research article. However, it’s worth being aware of the phrase “vibe-laundering” - the world is increasingly hard to parse and it’s challenging to be consistently diligent. The high level take-away is that an investment firm wrote an AI “think-piece” targeting companies they had short positions in and profited substantially. The market is being extremely narrative driven and it’s safe to assume similar “talking your book” disguised as AI commentary cropping up again and again.

Subscribe now

There are a few reasons to grade predictions. One is simply to have at least some form of accountability. If I consistently make predictions that are way off base, then my claims should be taken less seriously over time. And for myself, I think it’s a good way to learn. It’s really easy to remember things from even a few months ago as different than they were and having a written record of my thought processes is a good forcing function to confront the rapid rate of change. I’m only grading predictions that have a reasonable resolution at this point.

Then the second half of the post will be looking forward and making new predictions.

I used this prompt with Opus 4.6 to grade all the AI predictions I’ve made so far:

Please do a thorough search for predictions i’ve made on in this project.

Then figure out which of them have a resolution. For those that have resolved, grade them for quality. Then group them by performance.

For unresolved predictions, choose pragmatically to either give an indication of how it is going or note that it is simply unresolved.

Not all predictions were likely well operationationalized. If that is an issue, make a good faith effort to attempt to operationalize it and just be explicit about the assumptions you are making. If success or failure depends on one of these assumptions, its fine to mention that and give conditional grades depending on those details.

Any questions before getting started on this project?

(beginning of Claude’s output — lightly edited)

Scope: All identifiable predictions from project blog posts (early 2024 through late 2025)

Methodology

Predictions were extracted from across the blog corpus. Many were not originally written as formal, falsifiable predictions — they were embedded in analytical prose with hedging language. Where operationalization was ambiguous, I’ve stated my assumptions explicitly and provided conditional grades where the outcome depends on interpretation.

Grading Scale: 1–10, where 10 = nailed it (both direction and magnitude), 1 = confidently wrong. I also note prediction quality (was it well-operationalized?) separately from outcome accuracy.

You already self-graded three predictions in the GPT-5 post. I’ve included those with commentary on whether I agree with your self-assessments.

RESOLVED PREDICTIONS — Grouped by Performance

Strong Hits (7–10)

1. Impactful AI Agents by End of 2025

• Source: Reflecting and Projecting (Dec 2024), updated from May 2024 original

• Prediction: “We’ll have some version of actually impactful AI agents before the end of 2025 or if they do come out at the end, they will be better than I expected. ‘Meaningfully impact our lives’ is pretty vague, but I think we’ll know it when we see it.”

• Your Self-Grade: 9/10

• Outcome: Claude Code, Anthropic’s coding agent, became a genuine productivity tool well before end of 2025. Deep Research products from multiple labs. ChatGPT’s agentic features. Salesforce’s AgentForce handling 50% of customer interactions.

• My Grade: 8/10

• Reasoning: The direction was clearly right and the timeline was even faster than expected. I dock a point because the prediction’s vagueness (”we’ll know it when we see it”) makes it hard to have been wrong — this is more of a prediction quality issue than an accuracy issue. The prediction was right but not very falsifiable.

2. Codex-CLI Replaced by Something Better

• Source: Interviewing Software Engineers (mid-2025)

• Prediction: “I do not expect to be regularly using codex-cli in 3-6 months. I just think there will be something better.”

• Your Self-Grade: 10/10

• Outcome: Claude Code is indeed better, and you switched.

• My Grade: 9/10

• Reasoning: Accurate, well-scoped, and on timeline. The only reason it’s not a perfect 10 is that predicting a better tool will arrive in a fast-moving space is a relatively easy call. Still, you specified a tool and a timeframe, which is more than most people do.

3. Customer Support as First Major Industry Disrupted

• Source: Reflecting and Projecting (Dec 2024)

• Prediction: “The first industry that will be hugely disrupted (and it will happen in 2025) is customer support agents in call centers. At least one firm will do major layoffs in this area and replace/supplement workers with AI systems.”

• Outcome: Salesforce cut 4,000 customer support roles (from 9,000 to 5,000), explicitly citing AI agents. Sky reduced thousands of contact center roles. Klarna had already replaced 700 CS employees with AI. Amazon’s 14,000 corporate layoffs cited AI efficiency. AI was responsible for ~55,000 US layoffs in 2025 per Challenger, Gray & Christmas. Customer support was consistently identified as the first category hit.

• My Grade: 9/10

• Reasoning: This was specific, directional, and timed correctly. The Salesforce case alone would resolve this prediction affirmatively, but the breadth of examples makes it even stronger. This was a genuinely good call.

4. Customer Support Sub-Prediction: Customer Satisfaction Goes Up, Not Down

• Source: Same as above

• Prediction: “Customer satisfaction will go up not down”

• Outcome: Salesforce claims 94% of customers interacting with AI choose to continue using them voluntarily, with satisfaction scores exceeding human agents. However, this is complicated — Commonwealth Bank of Australia reversed course after workloads increased, and Gartner predicted half of companies that cut CS staff due to AI will rehire by 2027. Klarna also reversed some cuts.

• My Grade: 7/10

• Reasoning: The early returns from the successful implementations do show improved satisfaction, and this is directionally correct. But the picture is more mixed than you predicted — some companies overshot and had to walk it back. The prediction assumed cleaner success than actually occurred.

5. Less Emphasis on Pre-training, More on Test-Time Compute and Algorithmic Improvements

• Source: Reflecting and Projecting (Dec 2024)

• Prediction: “Less emphasis on pre-training, more emphasis on test time compute. More effort and results in other algorithmic improvements.”

• Outcome: The reasoning model paradigm (o1 → o3 → o4-mini, DeepSeek-R1, Claude’s extended thinking) became dominant. RL-based approaches proliferated. “Thinking” models became standard product features. Meanwhile, pure pre-training scaling received less hype (though it didn’t stop — GPT-5 shipped, Gemini 3 shipped).

• My Grade: 8/10

• Reasoning: Directionally very right. The nuance you added — “I am also not fully convinced that there aren’t still gains to be had in pre-training, I do not think a narrative that this is a dead end is either useful or likely to be true” — was also correct, as GPT-5 and Gemini 3 showed continued pre-training gains. Good calibration.

6. Pre-training Still Has Gains (Hedged Sub-prediction)

• Source: Same as above

• Prediction: “I am also not fully convinced that there aren’t still gains to be had in pre-training, I do not think a narrative that this is a dead end is either useful or likely to be true”

• Outcome: GPT-5 shipped. Gemini 3 shipped. Both showed meaningful gains from new pre-training runs, even if the “just scale it” narrative was more complex.

• My Grade: 8/10

• Reasoning: This was a good hedge against the prevailing “scaling is dead” narrative. Points for correctly identifying a nuance that most commentators missed.

7. Organizational Inertia as Primary Bottleneck

• Source: Reflecting and Projecting (Dec 2024), echoing March 2024 writing

• Prediction: “The main protective barrier we continue to have from overly rapid change and disruption is simply our deeply risk-averse incentive structures and the glacial pace of organizational change.”

• Outcome: Despite enormous AI capability improvements through 2025, most organizations adopted incrementally. Gartner found only 20% of CS leaders actually reduced staffing due to AI. The gap between what’s possible and what’s deployed remains enormous.

• My Grade: 9/10

• Reasoning: This was a structural insight, not just a prediction, and it held up extremely well. The pattern of “technology is ready but adoption lags” described 2025 very accurately.

8. Widely Popular AI-Generated Song

• Source: Reflecting and Projecting (Dec 2024)

• Prediction: “I think it’s a question of when, not if, there will be a widely popular AI generated song. There’s just too much latent creativity in the world that is enabled by these tools.”

• Outcome: Breaking Rust’s “Walk My Walk” hit #1 on Billboard’s Country Digital Song Sales chart with 3.5M+ Spotify streams. Xania Monet signed a multi-million dollar record deal. At least six AI artists charted on Billboard in 2025. Recording Academy CEO Harvey Mason Jr. said “every” songwriter and producer he knows has used AI tools.

• My Grade: 8/10

• Reasoning: This was framed as a “when not if” prediction without a specific timeline, which makes it less impressive as a prediction per se. But the speed at which it happened (within the same year) is notable. The direction and mechanism (latent creativity enabled by tools) were exactly right. A Deezer study found 97% of people can’t distinguish AI from human music.

Moderate Hits (4–6)

9. AI Products Begin Rapidly Evolving / Chat Interface Shift Begins

• Source: Reflecting and Projecting (Dec 2024)

• Prediction: “People will still think primarily of chat interfaces as AI, but this will begin shifting in 2025 and by 2026 AI will be pervasive enough that the predominant way of using AI will not be through chat / co-pilots”

• Operationalization assumption: For the 2025 portion, the question is whether the shift began. For the 2026 portion (that chat is no longer predominant), this resolves later this year.

• Outcome: Claude Code (terminal-based), Deep Research, Claude for Financial Services, various AI agents, and AI-native workflows all represent non-chat modalities that gained real traction. But as of Feb 2026, chat interfaces (ChatGPT, Claude.ai, Gemini) remain overwhelmingly how most people interact with AI.

• My Grade: 6/10 (for 2025 portion)

• Reasoning: The shift began — that’s fair. But the pace is slower than implied. The 2026 prediction that chat will no longer be “the predominant way” looks unlikely to resolve cleanly by year-end. Chat is still dominant by a wide margin.

10. Hiring Methodology Changes at Major Tech Companies

• Source: Reflecting and Projecting (Dec 2024)

• Prediction: “At least one, but certainly not the majority of major tech companies dramatically changes hiring methodology in light of AI. For example, allowing the use of models during an interview...”

• Your mid-2025 check-in: “I’d be happy to be wrong about this, but if it’s happening, I certainly haven’t heard anything even plausibly in this direction.”

• Outcome: By late 2025, the landscape shifted meaningfully. Meta piloted AI-assisted coding interviews. Anthropic permits Claude use for interview prep. Multiple startups are using AI-native interview processes. Work trials gained traction. However, the dominant pattern at Big Tech was the opposite direction — harder questions to combat AI cheating, return to in-person rounds, and 81% of Big Tech interviewers suspecting candidates of using AI.

• My Grade: 5/10

• Reasoning: This is a tough call. The letter of the prediction (”at least one major tech company dramatically changes”) is arguably met by Meta’s pilot and the broader industry shift toward testing AI collaboration skills. But the spirit of the prediction — that companies would embrace AI-friendly interview processes — is only partially borne out. The dominant response was defensive (anti-cheating measures), not the progressive embrace you envisioned. Your self-identified sub-prediction that “this will still not be the norm” and “I’ll still complain about how tech hiring is at an all-time high of being non-representative of the actual job” was correct.

11. National Security Risks Taken More Seriously, But Not Enough

• Source: Reflecting and Projecting (Dec 2024)

• Prediction: “People begin taking the national security risks of AI more seriously, but not enough to meaningfully do anything. There is still a window to act in 2026, but we really really need to take it then.”

• Operationalization assumption: “More seriously” means increased policy attention and discourse. “Not enough to meaningfully do anything” means no binding legislation or major institutional changes.

• Outcome: The AI labor policy discourse exploded (Obama and Bannon both flagging AI job loss as major political issue). The Keep Call Centers in America Act was introduced. EU AI Act implementation continued. But no transformative US legislation on AI national security passed. The policy conversation is louder but action remains largely incremental.

• My Grade: 6/10

• Reasoning: Directionally right, but this prediction was structured to be almost unfalsifiable — “more seriously, but not enough” covers a very wide range of outcomes. The prediction would have been stronger if it specified what “meaningfully do anything” looks like.

12. Step-Function Release Later in 2025

• Source: Gradually, Then Suddenly (mid-2025)

• Prediction: “I do expect a jarring step-function level release sometime later this year. Perhaps GPT5, perhaps an agent, maybe whatever is next from Anthropic.”

• Your Self-Grade: 2/10

• Outcome: GPT-5 launched but was more evolutionary than revolutionary. No single release created the “jarring” feeling you were predicting. Claude Code was the closest contender but was more of a product than a model release.

• My Grade: 3/10

• Reasoning: I’m slightly more generous than your self-grade because I think Claude Code + Veo 3 + the cumulative effect of 2025 releases did create significant capability jumps. But you were specifically predicting a single jarring release, and that didn’t happen. Your own analysis of why (pressure toward incremental releases, RL not extrapolating from o1 to o3 as expected) is sound. Good self-awareness here.

Misses or Too Early (1–3)

13. Video Generation as Biggest Miss / Timeline Update

• Source: Reflecting and Projecting (Dec 2024), updated in Updating on AI News (mid-2025)

• Original prediction: Sora was disappointing, updated timelines to video as hobby 3-5 years out (2027-2029)

• Revised prediction after Veo 3: “I’ll over-correct again to 2026 as the year I’ll release my first short film.”

• Outcome: TBD — it’s only Feb 2026. Veo 3 + Flow did represent a major leap.

• My Grade: N/A (unresolved), but the original over-correction from “2025” to “3-5 years” and then back to “2026” is actually a reasonable example of updating on evidence, even if you called it over-correcting.

META-ANALYSIS

Overall Calibration

Across the ~13 resolvable predictions, I’d characterize your record as meaningfully above average. Your strong suit is directional accuracy — you consistently identified the right trends (agents, customer support disruption, test-time compute, organizational inertia). Your weaker area is timing and magnitude — the step-function release miss and the chat-interface shift being slower than expected.

Prediction Quality Issues

Several of your predictions suffer from the hedging problem you’ve already identified:

• “More seriously, but not enough” (national security) — covers too wide a range

• “We’ll know it when we see it” (impactful agents) — unfalsifiable

• “Question of when, not if” (popular AI song) — no timeline = no risk

The predictions that were most impressive were the ones with specific, falsifiable claims: customer support as first industry disrupted, codex-cli being replaced, pre-training not being dead.

Pattern: You’re Better at Structural Predictions Than Timing

Your best calls are about what kind of thing will happen (customer support first, agents important, organizational inertia as bottleneck). Your weaker calls are about when exactly (step-function release, chat interface shift timeline). This makes sense — structural analysis draws on your comparative advantage in systems thinking, while timing requires predicting the decisions of small numbers of people (your own “idiosyncratic risk” framework).

Suggestion for Future Predictions

For your 2026 predictions, consider:

1. State confidence levels explicitly (you already do this sometimes — the 80% on legibility is good)

2. Specify resolution criteria in advance (”I will consider X confirmed if...”)

3. Separate directional predictions from timing predictions

4. Include at least one prediction you expect to be wrong about, to demonstrate calibration range

(it’s me rather than Claude again)

Previously I would have had an AI model do that grading merely as a way of indicating what is possible and showing the trajectory of capabilities progress. But now it’s just the right tool for the job. It’s simply faster (a few minutes instead of a half a day), more objective (although not without bias) and better than self-grading. It’s shocking to me how normalized this capability has become.

New Predictions

I’m starting with this one because even though it ignores all the lessons I just learned, it’s the most important one.

I cannot emphasize this enough. I’ve done what I think is a huge amount of effort to be emotionally prepared for the changes AI will bring and I am woefully unprepared. So chances are high this applies to you too. Let’s try to operationalize this with a set of baseline polls now that we can revisit at the end of 2026. I think the short time frame here is useful to consider, but where I’m extremely confident is this effect being felt widely by 2030.

I also took the 2026 AI Forecast survey:

My big swings are on revenue, remote work, and developer productivity. I realize that the revenue estimates exceed the companies’ own public estimates. But I also think that sometimes they might tamp down their projections in order to not seem crazy. I’m estimating notably more progress in remote work not because it is easy, but because the incentives for progress here are extremely high. The only area I’m more bearish than the median is on FrontierMath Tier 4 — these are the really hard problems in that benchmark. I think progress will be made there, but it will just still be a bit slow in 2026. I predict that we will have a level 3 (significant advance according to the framework here) AI math proof by:

• End of 2026 — 15%

• 2027 — 35%

• 2028 — 50%

• 2029 — 60%

• 2030 — 75%

This is an important one, because one of the biggest differences between my views and people less bullish on AI capabilities is exactly this issue of whether AI can produce novel and industry advancing insights. It’s clear to me that it can, but if we don’t begin to see evidence that this actually happens in the next handful of years, then I’ll need to substantially revise my worldview.

It’s also worth calling out the AI’s Societal Effects poll as a way of measuring AI backlash. If anything I think my estimate of net -25 is a bit conservative and I expect it to be substantially higher in 2027 (-40?).

One that I would update at least a bit already based on what’s transpired in the first two months of the year is the METR Horizon Doubling Time. Maybe Opus 4.6 is an outlier, but hitting 14 hours already sure makes it seem like we’re closer to a four month doubling time than the seven month doubling time. And maybe even faster — it was six hours just back in December. And then we’re less than two doublings away from models being able to succeed half the time on software engineering tasks that would typically take a human an entire week. For at least some tasks I’m currently doing, I think we’re already there.

I’m intentionally showing the linear graph instead of the log scale because I think it has more emotional impact for most people.

I predict sustained multi-quarter unemployment in the US of 6% or higher:

• 10% chance in 2026

• 35% by 2027

• 55% by 2028

• 75% by 2029

• 90% by 2030

In the long term, the major threat to this prediction is a large scale government make-work program. I very much hope we aim for UBI or similar strong social safety nets apart from make-work, but I’m not super optimistic there. One of the reasons I’m confident about this trajectory is the potential for feedback loops here. A relatively smaller change in one industry could easily cascade into several others. So the initial shock to get us there might be much smaller than is often considered. This could mean that substantial AI disruption in just one or two verticals could be sufficient to lead to these outcomes. Here is an AI analysis of this situation.

I’m not sure how to operationalize this one, but I’m strongly convinced that the gap between AI skeptics and AI realists continues to widen. This in part due to the continuation of high brow misinformation.

If you have any specific questions or areas where you’d be curious what I think, let me know and if I think I have any useful insight, I’d be happy to share it.

My general takeaways for 2026 are:

• the rate of change continues to increase (which is incredibly unsettling and destabilizing)

• more areas outside of coding will begin to experience what coding went through in late 2025 / early 2026

• real world impact will continue to lag capabilities due to diffusion, incentives, and organizational inertia in 2026, but in 2027 we will begin seeing undeniable changes in how the world works due to AI adoption

Best of luck to us all.

Subscribe now

The idea that coherence is truth conducive is intuitively appealing. Consider the case of eye-witness testimony. If there are three witnesses and they all agree about something, in general we will think the thing they agree on is more likely than if two of them agree and a third disagrees. Formal theories of coherence take this intuition and attempt to use math to make explicit just how much of a difference this makes.

An important caveat here is that for this to work, the individual witnesses must have some amount of reliability. If the witnesses have all agreed ahead of time to lie in the same way about the events, then the fact that they agree doesn’t make things any more likely.

The central idea to this research project has been that while LLMs do hallucinate, they have at least some reliability. So if we use LLMs to evaluate coherence, that coherence score will be associated with underlying reality in some way such that it can be used to reduce hallucinations.

The contributions of this work are:

• Validating theories of coherence as a project within formal epistemology through experimental philosophy.

• Demonstrating that coherence based approaches can reduce hallucinations by trading off a small number of correct answers for many fewer incorrect answers.

• A call to action to improve benchmarks which creates an incentivize for model providers to solve this problem in a more efficient way.

Let’s unpack the details.

Formal Theories of Coherence

A Probability Primer

In order to understand formal theories of coherence, we need to have a grounding in basic probability theory. Reading the more complex formulas for formal theories of coherence is easier if the individual foundational building blocks are clear.

Notation

These can be combined in many ways and they obey important axioms like:

Let’s go through a handful of examples the roll of a six-sided die to help make the abstract more concrete.

Given odd result, sample space is {1, 3, 5}. One of three is a 3.

A roll cannot be both 3 (odd) and even simultaneously.

Other Notation

The summation operator ∑ is more widely used and known than the product operator ∏. And the conjunction ⋀ and disjunction ⋁ operators are less common still. They are convenient shorthand that make it easy to express some common ideas in math, but if you aren’t familiar with them, they can make things seems more complicated than they are.

Overview

In Bayesian epistemology, coherence refers to how well a set of beliefs or propositions hang together in mutual support. Unlike mere logical consistency (a binary property), coherence is typically treated as a graded notion—some sets of propositions cohere more strongly than others. The formal measures below attempt to quantify this intuitive concept using probabilistic models.

Shogenji

Shogenji’s1 measure captures the idea that a coherent set of propositions should be positively correlated -they should “go together” more than chance would predict. The measure compares the actual joint probability of all propositions being true against the probability we would expect if they were statistically independent. When propositions mutually support each other (learning one is true raises the probability that the others are true), the joint probability exceeds the product of individual unconditional probabilities, yielding C_S​ > 1. This formalizes C.I. Lewis’s (1946) intuition that in a coherent set, each member should be supported by the conjunction of all the others.

Olsson

Olsson’s2 measure conceptualizes coherence as agreement or overlap rather than mutual support. It compares the probability that all propositions are true (the conjunction) against the probability that at least one is true (the disjunction). This ratio captures how much the propositions “say the same thing.” If they are equivalent, every world where any is true is a world where all are true, yielding C_O = 1. The measure treats coherence as the extent to which multiple pieces of information converge on the same content. Unlike Shogenji's measure, Olsson's is bounded between 0 and 1, which seems convenient. Higher values indicate a greater degree of overlap or agreement. But it lacks a meaningful threshold — independent propositions can score anywhere from near-zero to near-one depending on their base rates. Shogenji's measure always scores independence as exactly 1, giving us a clear reference point.

Fitelson

The Kemeny-Oppenheim confirmation measure:

Let’s unpack this:

• The numerator asks: how much more likely is K when H is true versus when H is false?

• The denominator normalizes this to keep the result bounded.

What F(H,K) tells you:

• F(H,K) = 1: H maximally confirms K (if H then certainly K; if not H then certainly not K).

• F(H,K) = 0: H is evidentially irrelevant to K (learning H doesn’t change your credence in K).

• F(H,K) = −1: H maximally disconfirms K (if H then certainly not K).

Fitelson’s3 measure then is:

Fitelson’s measure is grounded in confirmation theory — it quantifies coherence as the average degree of mutual confirmation among all subsets of propositions. Using the Kemeny-Oppenheim measure F, which compares the probability of a hypothesis given evidence versus given the negation of that evidence, C_F captures how strongly each part of a belief set confirms every other part. Unlike Shogenji’s simpler ratio, Fitelson’s approach considers all possible divisions of the set into “evidence” and “hypothesis” averaging confirmation scores across these partitions. This makes C_F sensitive to asymmetric support relationships and complex interdependencies. The negative range allows it to indicate not just absence of coherence but genuine conflict. Mutually contradictory propositions yield C_F = -1 rather than just zero.

Recap

Shogenji and Fitelson both capture coherence as mutual support, but Fitelson’s measure is more complex — averaging confirmation across all subset partitions and allowing negative values for conflict. Olsson’s measure departs from the confirmation paradigm, focusing on overlap rather than evidential support. These differing conceptualizations mean the measures can disagree on which of two sets is more coherent.

Example

Let’s take a simple example and apply all of our coherence measures to it.

• H₁: The butler was at the manor last night.

• H₂: The butler’s fingerprints are on the safe.

• H₃: The butler stole the diamonds.

And we’ll assign the following probabilities:

The butler is often at the manor at night, but it is unusual to have fingerprints on the safe and even more rare for the butler to steal the diamonds.

The pairwise probabilities of each of these (at the manor + fingerprints, at the manor + theft, fingerprints + theft). Note that if these events were fully independent these probabilities would be different. For independent events the P(A ^ B) is P(A) * the P(B) — in this case 0.6 * 0.2 = 0.12. But for dependent events the P(A ^ B) = P(A) * P(B|A) — in this case 0.6 * 0.25 = 0.15. There is sometimes yet another calculation that I’m not explicitly demonstrating. This diagram provides a visual representation:

The probability all of them combined:

The probability if all these events were fully independent of each other.

Shogenji’s formula with the variables:

and calculation:

Olsson’s formula:

denominator disjunction:

calculation:

and we already know the numerator conjunction as 0.07 so:

For Fitelson, there are six partitions that satisfy the criteria. Remember that each partition must contain two splits. Each split must be non-empty and the splits together must contain all the propositions. Importantly, these are also ordered pairs, so S₁ = {H₁} and S₂ = {H₂, H₃} is different from S₁ = {H₂, H₃} and S₂ = {H₁}.

Let’s cover the setup and calculation for just one of the splits since it’s rather involved, but the other splits are done in a similar fashion.

Does knowing the butler was present H₁ confirm that both his prints are on the safe AND he stole the diamonds H₂ ^ H₃? To calculate the Kemeny-Oppenheim measure we need two conditional probabilities:

If there butler was present, there is an 11% chance the butler left prints on the safe AND stole the diamonds.

And then if the butler wasn’t present, there is only a 2.5% chance (perhaps the diamonds were stolen before last night).

Those values can then be plugged in to calculate:

So the butler being present does provide some evidence for having stolen the diamonds and left prints on the safe, which is an intuitive result here.

The final calculation of all six splits is:

Coherence Experiments

This research has two aims. One is philosophical and one is pragmatic. But in the support of the goals of the these are related. Can theoretical ideas aid us in truth-seeking?

The philosophical goal was in the spirit of experimental philosophy. It is common for philosophers to assert that a particular view is intuitive. But is it? Experimental philosophy attempts to use empirical methods to understand what beliefs people actually have. An example of this is my work on analyzing Kiva micro-finance data to understand how people think about aiding others with the hope that this can constructively inform discussions of ethical theories.

The dramatic improvements in AI provide a new avenue of exploration for experimental philosophy. The formal theories of coherence outlined above have deep theoretical motivation but limited empirical validation. By using AI, we can empirically answer the question of whether these formulations are indeed truth conducive and whether any of them have advantages over the others.

The pragmatic goal was to attempt to use coherence to reduce hallucinations in AI responses. Hallucinations are one of the most important limitations of current AI models. Making up citations, fabricating evidence, or simply being confidently wrong are all outcomes that dramatically limit the utility of AI systems — reducing reliability, undermining confidence, and increasing the importance and cost of verification. AI has been so heavily incentivized to get an answer correct at all costs that it fails to say “I don’t know” when it should. Humans also struggle with this.

Experimental Philosophy with SelfCheckGPT

To investigate the empirical effectiveness of formal theories of coherence, I build on the closely related work done with SelfCheckGPT.

SelfCheckGPT leverages the simple idea that if an LLM has knowledge of a given concept, sampled responses are likely to be similar and contain consistent facts. However, for hallucinated facts, stochastically sampled responses are likely to diverge and contradict one another.

The core intuition and motivation here is the same, so the work to be done is to operationalize the formal theories of coherence within the framework setup by SelfCheckGPT.

SelfCheckGPT uses the WikiBio dataset which is biographical data scraped from Wikipedia. Then an LLM was used to generate statements about the individuals in the dataset and another LLM is asked whether the sample supports the generated samples. A yes is 0, a no is 1 and N/A is 0.5 (I found this inversion unintuitive, but the idea is that higher scores are more likely to be hallucinations.) These answers are then compared against the manual annotations done in constructing the dataset.

The coherence based approach4 here is to take the sample data and all the generated statements and feed them into our various coherence formulas. The formulas require probability estimates, which are made using an LLM. The coherence scores5 are then normalized6 to fit the same range, scale, and semantics.

SelfCheckGPT contains many other methods, but the LLM prompt using gpt-3.5-turbo produced the best results. The data indicates the coherence measures being competitive with that result:

• Nearly as good as the benchmark on non-factual samples (NonFact AUC-PR).

• Notably better on factual samples (Factual AUC-PR).

• Much less well correlated with human judgements (PCC).

There are two major takeaways from this experiment. First, this is a vindication of the general project of formal coherence theories. If the mathematical formalism here was not connected with reality, we would obtain far worse results here. We’ve replaced a very simple scoring approach of 0, 0.5 and 1 derived from a prompt of an LLM model with a much more complicated calculation. Mere chance cannot explain these results. Formal theories of coherence may be pragmatically useful in addition to being theoretically interesting.

A second takeaway is that much of the coherence literature is focused on the exact formal specification, arguing for one measure over another based on theoretical criteria. I did the experiments with all three coherence measures in the hope that we might empirically validate one of the approaches. Experimental philosophy is rooted in the idea that empirical work can inform theory. I had hoped that we might be able to have data clearly indicating that the increased complexity of Fitelson’s approach was justified or that the overlap approach of Olsson that I favored back in graduate school would prevail.

But, as you can tell from the table above, the results for each approach are clustered tightly together. Yes, Fitelson’s measure has slightly better precision, but it’s also less correlated with human judgement. Benchmark interpretation is not always straightforward, but these small differences do not seem to be sufficient evidence to meaningfully inform theory. One benchmark and experiment isn’t decisive in settling any long-standing debates. These results gesture at the idea that they are all capturing something but with none decisively better or worse than any other. This suggests that the field does not need yet another formalization or critique of an existing measure. Instead, now that it is possible to do experiments with these theories, it would be far more productive to do more applied work. The second experiment in this project is one such attempt to pragmatically apply coherence to one of the most pressing problems in AI: hallucinations.

Pragmatic experiments with SimpleQA

It was a bit challenging to determine how to apply coherence to other benchmarks, but it was worth considering how to generalize the approach so that it could be used in other situations. There are many benchmarks in the hallucination space and my early research in the project focused on attempting to reproduce them. I focused on SimpleQA for a few reasons:

• I was able to fully reproduce the benchmark results.

• There were no API limitations such as requiring the log-probs output from the response.

• It is a high quality dataset along with being intuitive and easy to understand.

However, there were still some challenges in adapting the coherence work to SimpleQA. Two facets of the structure of SimpleQA made this not straightforward. First, the prompt component of SimpleQA is a question. Questions don’t have truth values, they’re just questions. The idea of coherence can’t be applied to a question the same way it can to a series of facts or claims. Second, the answers are quite short — just concise answers to the question. And coherence doesn’t work that well on short claims. If we just have a single statement, this reduces to the P(A) and coherence doesn’t help if there are not other statements for it to cohere with.

So, areas where coherence may more naturally fit without additional effort are if the prompt contains a lot of additional context and information that coherence could apply to or if the answer itself is sufficiently complex and contains enough statements.

The approach7 I took was to generate variations of the question, have the model sketch out answers to these questions using different temperatures, and then apply coherence to the set of answers that was generated. One of the core ideas was that we could use different model temperatures to produce increasingly varied results. Coherence across lower temperature responses and higher temperature responses would then be an even stronger signal of being grounded in reality.

Unlike in the SelfCheckGPT benchmark where the coherence score itself had semantic meaning (being treated as roughly equivalent to the probability of a yes or no) here we still need the model to produce a response. So, I used coherence as a threshold. If the coherence of the answer variations was sufficiently low, I had the model abstain from answering the question. The baseline prompt for SimpleQA does give the model the option to refrain from answering in this way and it does this occasionally, but not nearly as often as desired.

AI benchmarks in general have been over-optimized for correct answers at all costs and not sufficiently penalized for incorrect answers. In many cases, increasing the number of “I don’t know’s” is highly valuable. The SimpleQA results are predictably reported as purely the percentage of correct answers. The approach I’ve taken cannot possibly increase the number of correct answers. That is not where the value lies. The value is in refusing to answer when the model is less confident because the coherence score is too low. Of course, we want to preserve as many correct answers as possible. Here are the results comparing the baseline SimpleQA approach to my coherence approach8:

We modestly improve the ‘correct given attempted’ result, while dramatically reducing the number of incorrect answers. This does come at a cost because the increase in ‘not attempted’ does mean the models could have answered some questions correctly but didn’t.

These handful of correct answers lost are such a small price to pay for the reduction in incorrect answers. The worse the model is or the harder the domain the greater the benefits. GPT-4.1 is a much better model for this task with a score of 41.6 vs gpt-4o-mini scoring just 9.5. So the benefits to this approach for the smaller model9 are even more pronounced, but avoiding five wrong answers per correct answer lost is still a compelling tradeoff even for the more capable model.

Conclusion

The truth matters. Dramatic advances in AI coinciding with a cultural tendency to embrace a post-factual mindset is perilous. We should be heartened that philosophy is as relevant as ever. What lessons can we learn from this investigation?

AI and Hallucinations

Hallucinations are a critical threat to the usefulness of AI models. From a practical standpoint, they increase the cost of verification. And from a more psychological and emotional perspective they erode trust. In the worst case scenarios they can lead to catastrophic failures where a user trusts the results, but does not verify the results despite the stakes being high. It’s easy to say that people should always do verification when the stakes are non-trivial, but in practice this simply won’t or can’t always be the case.

The model providers know this is an issue and continue to make progress on this front, but more is needed. While the coherence research here provides a potential mechanism to improve on this, it’s ultimately not practical for a few reasons. One is that it adds so many more API calls and token consumption. What would have been a single call to an LLM could easily become dozens. This adds cost and increases the time to a result. Another is that it took a bespoke effort to adapt coherence to work with this benchmark and similar efforts would be required for other use cases — it’s powerful, but doesn’t easily generalize10. So instead of advocating for the adoption of coherence based techniques, it’s enough to know that substantial progress is possible here and to look for more effective means of accomplishing the same goals.

The simplest and most straightforward thing to advocate for is structural change in benchmark creation and benchmark results reporting11. Wrong answers have a cost. Back when capabilities were so much more limited, it was sufficient to only focus on getting any correct answers. But now that the models are so much more powerful, we must have benchmarks that not only reward correct answers, but appropriately penalize guessing and hallucinations. Perhaps a cost of verification and the damage from accepting an incorrect answer can become standardized components of evaluations.

Similarly, a lot can be learned from how ARC-AGI presents results — in addition to the score, each result is reported with the cost per task. This framing was implicit when I argued against the adoption of coherence based measures for solving this problem. Increasing the cost by an order of magnitude or more for some improvement can sometimes be worth it, but for many tasks will not be. As we think through the tradeoffs of using AI more widely cost will become an increasingly important component.

If the prominent benchmarks were to appropriately represent the costs of wrong answers and encourage saying “I don’t know” consequences would easily flow downstream. Model creators would become strongly incentivized to perform well and would come up with better12 and better ways to accomplish this. This follows from the normalization of including benchmark results as part of each release. It is extremely nuanced and difficult to know whether one model is better than another model and like so many things it depends. It depends on the use case, the skill of the user, the scaffolding, and is constrained by issues like cost, latency, reliability or even the ergonomics of the API design. Model providers want to be able to declare victory that their model is the best. Benchmarks provide them with a small number of supposedly comparable numbers, which then simplifies this complex and nuanced set of tradeoffs into some basic math that yes 79 is indeed greater than 76. The attention to benchmarks and the value to model creators is so high that it’s common for there to be concerns about ‘benchmark maxing’ a model — training it in such a way that the numbers go up on the benchmark, but the real world performance lags behind. Which leads to another desirable consequence of closing some of the gap between the shockingly amazing benchmark scores and merely quite good real world applications.

Truth and the Future

I started this project with several goals in mind. Expanding the tool-kit of experimental philosophy using AI is a worthwhile endeavor all on its own. The kinds of experiments that can be done and the ways that we might learn are rapidly changing and becoming increasingly accessible. On top of methodological innovation, I wanted to understand if the project of formal epistemology was more than theoretically interesting. And I wanted to determine if hallucinations could be reduced using these approaches.

The SelfCheckGPT experiment accomplishes the experimental philosophy and formal epistemology goals by demonstrating that formal theories of coherence, while not advancing the state of the art on the WikiBio benchmark, are highly competitive and a reasonable approach. Coherence is truth conducive in a real world setting. This experiment would have been almost unthinkable decades ago. AI could usher in a new era for experimental philosophy.

And I was able to reduce hallucinations substantially on the SimpleQA benchmark using a coherence based approach. While it does not improve the raw accuracy, we can trade a small number of correct answers to get a large reduction in hallucinations. For the many use cases where a wrong answer incurs at least some cost, this is an excellent tradeoff.

While the philosophical line of inquiry is worth continuing to pursue here, I recommend against adopting this exact approach of coherence based measures as an implementation for reducing hallucinations. The experiment demonstrates that this is possible, but it is not ideal for a few reasons. If the approach could be evolved so that it was an efficient, simple wrapper that could be applied to any query, then it might be worth considering. But even then, there is a better way.

Ultimately what this investigation has convinced me of is the need to advocate for changes in the benchmarks themselves. This is a powerful lever that can be used to create compelling incentives for model creators. The core idea is that only counting correct answers and not appropriately weighing the costs of hallucinations in many of these popular benchmarks, we incentivize model creators to not prioritize this problem. If all mainstream benchmarks that get reported upon model releases include not just correct answers, but also penalize guessing or hallucinating, we would rapidly see both promising improvements in model behavior and benchmarks that are more reflective of real world experience and utility. While not the focus of this research, results should also be normalized by compute costs. The coherence based approaches I experimented with are not efficient, but model providers can work on this problem farther upstream and come up with more effective ways to solve the problem. If there’s one lesson we should be repeatedly learning about AI progress, it’s that what’s expensive and impractical can quickly become cheap and easy.

Truth isn’t just correct answers. It’s also knowing what is incorrect. Or in the absence of those, it’s the willingness to say “I don’t know”. In the most hopeful visions of the future, AI is something we can trust and rely on. One concrete step that we can take toward getting the truth we deserve is to create better benchmarks and demand that model creators take them seriously.

Subscribe now

I wanted to write an essay proclaiming some grand advice about how young people should approach AI differently than older people. But, turns out that I don’t have the overconfidence to bluster through such an exercise. No one knows. So instead, I want to offer a framing of the problem and simply acknowledge the difficulty without platitudes.

A historical precedent for epistemic humility here: social media. While the full impact of social media on society is still not understood, I think it’s safe to say that it has a lot of negative externalities1 that weren’t considered at all upon its creation and release. And many of these issues disproportionately impact young people. The impact of AI will be much, much larger than social media.

The problem

One of the issues is that there isn’t just one problem, but many inter-related problems. We crave simplicity, reduction, clarity and certainty. We struggle with uncertainty, nuance, complexity and vagueness. There is an almost primal instinct to consolidate the many into the one. Here’s an attempt to enumerate many of the problems:

• once-in-a-lifetime levels of uncertainty

• the speed of change

• reduction in value of instrumental knowledge

• domains shifting in importance

• a crisis in credentialism

• the ability to act as if you have knowledge without acquiring that knowledge

• navigating whether using AI is high status or low status in different contexts

• a demand to be AI-fluent

• an equal imperative to be able to do everything without AI

• a need to project out to decades when even five years is unclear

• challenging job market that is particularly hostile to entry-level positions

• mentors offering well-intentioned but harmful advice tied to a status-quo that no longer exists

• at best educators struggling to find meaningful pedagogy for this era in real time, at worst educators failing to change anything at all

• no line of sight to the long term impacts of AI usage

• contradictions exist within the problems and expectations with no clear resolution

I’m sure there are more (climate change, the rise of fascism, aging demographics…) but this is enough to gesture at the breadth and difficulty of the problems. It’s essential that we recognize just how challenging this situation is. It’s deeply patronizing to offer blanket reassurance without justification. But it’s also harmful to be a doomer and throw our hands up in exasperated despair.

Untangling all of these problems is beyond the scope of this essay, so I’m going to focus on the knot that is centered around knowledge and reasoning.

Knowledge and Reasoning

I grew up in a deeply anti-intellectual community. It’s the 100 year anniversary of the Scopes Monkey Trial. For many of you that holds little cultural relevance because it feels so settled and so long ago. But for me in my childhood, it was a battle that was still being fought and lost. I think it’s exactly these sorts of experiences that drove me to love knowledge and value capital ‘T’ Truth so much.

I share this context because it’s important to understand that knowledge and reasoning are not things that I hold lightly or take for granted. The reverence I have for ideas is as close to the spiritual as I’ll ever get. For decades a deep cornerstone of my identity was smart person who knows a lot of things. What I’m grasping for is the desire for credibility of what follows.

The erosion of the usefulness of instrumental knowledge

One of my core hypotheses about AI is that it changes our relationship in particular with intermediate level knowledge. AI provides us the ability to act as if we know the underlying details without us actually understanding them. It provides a higher level of abstraction to interact with ideas.

Let’s take a concrete example of a spreadsheet. You have to present some data at a meeting and want others to understand the data better. Previously, you’d need to know at least a few concepts (formulas, cells, etc.) and combine them together to create the spreadsheet you need. Some people know more concepts than others and maybe after seeing someone present a useful example, you ask them how they did it and you build up your spreadsheet vocabulary over time. But now, if you know what you want, you don’t need to know very much of the how at all.

As I’ve written about before, unless you’re a true expert, AI (at least Claude) is already better at spreadsheets than you are.

This is simultaneously great and an unsolved challenge.

It’s great because it democratizes access to the technology. More people can more easily solve problems and provide value in this area with less experience and knowledge. And people with existing knowledge can create more quickly and prolifically. If there was ever value in creating the spreadsheet, it has never been easier to do it.

But now we get to the tricky part. What does this capability do to our ability in the long-term to critically think and reason? I think this is an empirical question that we should strive to understand sooner rather than later, but I don’t think we’ll have satisfying answers for decades (how can we know how 20 years of AI use from age 5 to 25 impacts the brain until we have humans actually do this?). Since we don’t know, let’s at least model the possible worlds we could be in and use them as guides.

Possible World 1: The scaffold is necessary

In this scenario, it is the case that the only way to acquire genuine critical thinking skills is to do the hard work to learn something. Try to understand the thing that is just outside your current capabilities, sometimes succeed and sometimes fail — and keep trying. This is the classic scaffold of education and learning. And then as a byproduct of the struggle you gain a plethora of meta benefits: taste, discernment, judgment, learning how to learn.

Why is this world so grim?

Historically, the thing you were trying to learn provided you some substantial value. Learning to code has been an economically useful skill. The instrumental value from the intermediate knowledge provides substantial motivation. If my hypothesis of erosion of value of intermediate knowledge is correct then extrinsic motivation for struggling to learn most things plummets. And if motivation plummets, it seems so challenging for people broadly to put in the effort to learn things that don’t really benefit them just for the hopes of acquiring meta critical thinking skills.

I don’t think I could have sustained the motivation to have learned much of what I have learned if I hadn’t thought that it would be useful. And now that we’re past the point of AI models being better at coding than I am and on the cusp of them being as good at software engineering, I can’t imagine how someone starting out isn’t impacted by this. For context, I’m also highly intrinsically motivated. I have deep curiosity and a genuine love for ideas. But this is still how I feel.

Why is this so much worse for young people? I’ve already put in decades of effort. The old system worked for me. Yes, I’m still in danger of having skill degradation, but I’m aware of where it’s happening and can be intentional about the tradeoffs. The core of my critical thinking skills are well developed and intact.

Possible World 2: The scaffold is unnecessary

In this more optimistic scenario, it’s possible to acquire these essential judgment related thinking skills without striving to understand some no longer useful instrumental knowledge. Maybe we can study critical thinking and learn thinking skills purely on their own without the need to have the traditional scaffolding in place. These essential skills are no longer a by-product but the first class goal of education.

The upside of this scenario is that motivation can remain intact: the goal and the outcome remain related. The challenge here is that we’d need a massive innovation in pedagogy across the board. Everything in the current system is setup to learn a particular thing. The uncertainty here is daunting. Can this be done fast enough? Would it work at all? Will it be scalable and accessible?

But if we’re in this world and we take it seriously, then AI can actually be a huge boon to young people. If we have a generation that is highly skilled at asking the right questions, framing problems well, rejecting premises, and evaluating ideas — it could be transformative. If knowing a thing is never a bottleneck again, it’s actually a great equalizer.

Possible World 3: Even these meta cognitive skills are not economically useful in the age of AI

I refuse to believe that critical thinking skills have been over-rated. However, we must accept the possibility that even the most useful of human cognition might still not be economically valuable in the face of AI. In this scenario, regardless of how we help young people acquire skills around judgment and critical thinking, they are simply not above the threshold of economic value.

In this world, the entire framing is different. What human activities remain valuable and why?

This is the most polarizing scenario with the highest variance outcomes. It could easily be deeply dystopian and bleak. But it also allows the opportunity for tremendous flourishing. If we no longer had to struggle to survive and could focus purely on pursuits for their own sake, what would we do and what could we accomplish? One of the reasons I take leisure so seriously is a belief that it is a deep privilege to have any leisure time at all and it’s a way to honor the struggle of previous generations to enjoy leisure as much as possible.

How many more artists would we have? How many more shared meals with family and friends? What games might we invent? How much more time in nature?

So?

One of the practices that has made my life consistently better is the regular attempt to cultivate an attitude of optimism. I’m pretty naturally wired as a pessimist, but life is just more enjoyable and tends to work out better for me the more I experience things as an optimist. For me optimism is earned, but worth the effort.

So I don’t really have any concrete advice for young people, but I very much hope that things work out okay for them. But I think it’s important that we say and acknowledge that yes, this is hard and uncertain — no one can tell them what’s coming or how to prepare. Everything is changing; let’s work together to try to make it be for the better.

Good luck.

Subscribe now

(from the new Nano Banana Pro model — I asked for a thumbnail for this blogpost with no other context)

One of the first benchmarks I wrote about for this blog was ARC-AGI. About a year ago, o3 ushered in a new era with birth of the “reasoning model” paradigm and breakthrough scores on ARC-AGI. But there was a huge caveat. The cost per-task for this level of performance was prohibitively expensive for anything outside a highly funded research project or proof of concept. o3-preview cost about $200 per task to achieve 75% performance and a high compute version (using 172x as much compute) was able to achieve 87% performance. This comes to be 200 * 172 = $34,400 per task. While there was a lot of enthusiasm for the breakthrough, skeptics were rife with their commentary on the excessive cost. But the ARC-AGI folks understood:

But cost-performance will likely improve quite dramatically over the next few months and years, so you should plan for these capabilities to become competitive with human work within a fairly short timeline.

Gemini 3 is a shocking vindication of this claim. It’s about a year later and:

Gemini 3 Pro achieves a score of 75% at a cost of $0.49 per task. About a 400x improvement over 1 year.

Gemini 3 Deep Think achieves a score of 87% at a cost of $44.26 per task. Over a 700x improvement over 1 year.

You’ve probably detected a consistent pattern in my thinking: I expect fast and impressive improvements; I am surprised by just how fast and impressive these improvements happen.

It’s really convenient for such a comparison for the scores to have been so similar that it’s trivially easy to compare the cost in this way. Another way to look at this is how many tasks you could complete for a fixed budget. I think this framing is useful because it provides a more useful economic framing. Consider for a moment that these tasks had some economic value and that we might be able to estimate their economic value. Then if we understood how much it would cost to solve them, we could straightforwardly understand how much money we’d make (or lose) by solving them with AI. Obviously the ARC-AGI tasks themselves do not have economic value, but they are a crude proxy for some level of value.

Part of my continued conviction in the high likelihood of substantial disruption to the labor market is based quite simply on the idea that sometime relatively soon there will be many use cases for which the economic value of a task completable by AI is far higher than its cost to compute. And I’m actually quite confident we’re already in this scenario and it’s simply slow diffusion, risk aversion, status-quo bias, and other social and psychological factors that are holding back the torrent of radical change.

This is cost per estimated successful task. I do not include a cost for verification of correctness. This oversells the economic case for Gemini 3 Pro to some extent. A 75% success rate still means 1 out of 4 are failures. For some use cases this introduces a substantial cost of verification or risk mitigation. However, for many use cases either the cost of failure is low or verification is not too expensive. But a use case dependent analysis can then determine whether it’s better to get ~15,000 tasks completed for 10k dollars or 588.

A similar concern is raised in the GDPval benchmark that was released a couple months ago and has this exact purpose in mind.

In addition, we found that frontier models can complete GDPval tasks roughly 100x faster and 100x cheaper than industry experts. However, these figures reflect pure model inference time and API billing rates, and therefore do not capture the human oversight, iteration, and integration steps required in real workplace settings to use our models. Still, especially on the subset of tasks where models are particularly strong, we expect that giving a task to a model before trying it with a human would save time and money.

Claude Opus 4.1 is shockingly close to parity with industry experts —“These professionals averaged 14 years of experience, with strong records of advancement” on these selected tasks. I would be very curious to see results for GPT-5.1, Gemini 3 Pro and Gemini 3 Deep Thnk.

Imagine what $34,000 of compute used with today’s models and scaffolding (Claude Code, Codex, etc) compared to this time last year. But let’s also project forward a year. We might have comparable performance at 400-700x cheaper. Or we might have much better performance. Or maybe some of both. Far less than 400x gains would be required to make these knowledge worker tasks shockingly cost effective for AI.

A highly simplified view of advertising is that if the cost of customer acquisition is less than the lifetime value of the customer, you pump money into the system and get more back out. When the gap is close, you of course need to worry about the time value of money, risk, and other factors. But when it’s not close, you just print money while you can. GPDVal is the closest thing we have for estimating where we are in the value of AI labor. Right now, it looks pretty close. But if it’s close now, next year it won’t be. The implications of this are stark.

A few other ways to think about the extreme rate of progress

The qualitative leap is just as striking. The problem with benchmarks and numbers is that they are abstract and often fail to emotionally resonate. So here are a handful of ways to try to understand the same point being made above, but hopefully more salient and emotionally resonant:

The recent history of AI in 32 Otters by

Will Smith Eating Spaghetti (two and a half years ago vs recently)

We have the first multi-million dollar music deal for an AI Artist: Xania Monet. A Deezer study that 97% of people can’t distinguish between AI generated and human music. I was an early adopter of creating AI music and can assure you that I could tell the difference when making AI music in 2023 and 2024. (I have not been offered a multi-million dollar deal, for obvious reasons, but if you haven’t checked out AISlothArmy, now is your chance).

Is AI a Bubble?

My continued contrarian take here is simply no. (I’m contrarian compared to the general narrative in the zeitgeist. I do not think I am actually that contrarian from many in the AI establishment.) I find the trends outlined above too compelling and the Gemini 3 release is a strong data point that these trends are continuing. In addition to advancing frontier capabilities, Gemini 3 shows dramatic improvement in the cost effectiveness of capabilities that were state of the art just a year ago. Once capabilities can be taken for granted, then cost becomes the determining factor in the use of AI labor.

I see 2024 as the year when AI enthusiasts had to frequently say things like: “this is the worst the models will ever be” and “imagine what this will look like in 6-12 months”. Because while the results were impressive, it was much harder to get clear value directly. And so much of the value was located in the future, the promise of what would soon be.

But 2025 has been a tipping point in capabilities. I still say things like I used to last year, but it’s no longer to make excuses for current short comings. It’s more a reminder to myself that what’s to come is still such a big deal. Claude Code is just excellent. Will it be better in the future? Of course, but it’s already great. Deep Research, Nano Banana Pro, Sora 2, Notebook LLM, Claude for Financial Services, and music generation that is virtually indistinguishable for many listeners are all things in the present, not some future promise. Simply finding, collating, and processing information has never been easier or better.

I hope to do a New Year's post with fuller predictions and thoughts about 2026, but a few short thoughts here. Even a 10% probability of an extreme level of transformation from AI makes AI undervalued. I do not think that most people take the possibility of true transformative disruption seriously. My advice is to find a way to viscerally feel the rate of change for yourself and once you’ve had that experience, it will change how you look at things. (Check out some of the options in this section: “A few other ways to think about the extreme rate of progress” or even better use more of these tools yourself more often for use cases that are relevant to your life. ) Then prioritize for short term capital accumulation over other competing goals. If you’re established in your career, I think a lot of this advice still holds. If you’re much earlier in your career or still in school, I think things are lot more complicated — I hope to think and write more about this in the future.

Despite the fact that I continue to underestimate the stickiness of human status-quo bias and the challenges of coordination around new ways of working, I’m still convinced that substantial change is coming. Greed is too powerful a force to leave trillions of dollars on the table.

Subscribe now

Thanks so much to my paid subscribers, here is a special thank you for your support:

Read more

I’ve spent the first third of my time attempting to reproduce a handful of existing AI benchmarks in the hallucination space: SimpleQA, HaluEval, FaithBench, and TruthfulQA. The general mission of these benchmarks is well summarized by the OpenAI announcement of SimpleQA:

An open problem in artificial intelligence is how to train models that produce responses that are factually correct. Current language models sometimes produce false outputs or answers unsubstantiated by evidence, a problem known as “hallucinations”. Language models that generate more accurate responses with fewer hallucinations are more trustworthy and can be used in a broader range of applications. To measure the factuality of language models, we are open-sourcing⁠(opens in a new window) a new benchmark called SimpleQA.

My general intent was an attempt to be rigorous here. If I could reproduce the published benchmark1, then the coherence modifications to the same benchmark would have a high degree of confidence of actually being due to the novel approach rather than other factors. This sounded good in theory (and I think actually is good in theory). However, the term “reproducibility crisis” exists for a reason.

After a moderate amount of effort, I succeeded in replicating all results for SimpleQA; for HaluEval I came close to parity for some but not all of the evaluation metrics. FaithBench and TruthfulQA were not particularly close. I could have invested more time and energy here. I could have contacted the original authors. But, it wasn’t absolutely critical. I give this as context that these are not claims that a higher reproducibility score couldn’t be achieved, just that it wasn’t easy and the return on investment was rapidly diminishing. Especially since this was not the core focus of the grant.

The goal of this post is to outline some of the challenges I ran into with special emphasis on reproducibility of AI research and then to provide some suggestions for improvements for researchers working in this space.

Model Accessibility

Papers of course indicate which models they tested, but frequently do so in an underspecified way. A common model used in the research is gpt-3.5-turbo (because these ancient papers are from a year or two ago). But which model is this exactly? OpenAI still hosts these specific legacy model snapshots:

• gpt-3.5-turbo-0125

• gpt-3.5-turbo-1106

• gpt-3.5-turbo-0613

Along with a `gpt-3.5-turbo` model. But which model is actually being served at that endpoint now? Which model was being served when the researchers did their work? If they didn’t specify a snapshot version, could it have changed between runs of their analysis? And even if the researchers had been more precise and specified an exact snapshot, would that snapshot even be available now? When will OpenAI2 stop providing these models entirely?

And on top of all of this, it’s likely there are backend configuration parameters that could change over time. So even if you call `gpt-3.5-turbo-0125` today, it might not behave exactly the same as calling it last year or two years ago.

Here’s a concrete example. The wiki_bio_gpt3_hallucination dataset and benchmark contains sentences like these:

John Russell Reynolds (1820–1876) was an English lawyer, judge, and author.

gpt-3.5-turbo-1106 considers this factually supported, while gpt-3.5-turbo-0125 correctly susses out the inaccuracy (Reynolds was a physician and not a lawyer).

Why does this matter?

In some cases small differences can really compound. In other cases, they just introduce a lot of noise and uncertainty. And at the very least it makes full reproducibility almost impossible. A small but useful area of research could be to understand how large these differences could be — as a way of guiding researchers confronting this problem to understand if this could plausibly be the variation they are observing,

Fine-Tuned LLM as a Judge

LLM as a Judge is a common technique for use evaluation. While we might prefer more traditional metrics, often these metrics cannot capture the nuance and details that an LLM can. In order to get the desired results, sometimes researchers fine-tune an LLM specifically for this purpose and their use case. This makes sense and isn’t inherently problematic, but these fine-tuned versions are not easily accessible and even if the researchers clearly show how they did the fine-tuning, you’d then have to perfectly replicate the fine-tuning in order to replicate the results of the benchmark.

Random Seeds

One project used random number generation for a critical component but failed to specify a seed for the random number generation in the code. This means that their own internal runs would not have been reproducible, much less someone else trying to replicate.

Versioning

Not all projects specify the versions of code they used. There can be bugs or at the very least differences in output between versions of code. This can range from not including any dependencies or version information at all in the code repository to including some but not using pinned versions.

Porting

And I made things more difficult by trying to consolidate various benchmarks in one consistent framework. The appeal of this is high for functionality and convenience, but it undoubtedly introduced additional risk to reproducibility simply due to complexity, difference, and inevitable bugs. A lesson in over-confidence that will not be the last for me.

Temperature

This one wasn’t problematic for any of the benchmarks I tried to replicate, but it will be an issue going into the future. It is best practice in benchmarks to use temperature 0. This leads to nearly deterministic results from the model. However, most current reasoning models don’t support configurable temperature and always use a value of 1. This makes the output substantially more varied. But reasoning models are of course of great interest, so it’s critical that we find other approaches to improving reproducibility when using models that don’t support temperature 0.

Suggestions

Reproducibility continues to be a challenging problem. There are just so many things that a researcher can take for granted and also so much implicit context in individual human brains. But here are some ideas for directional improvement:

• If using the API of a model provider and it has a dated snapshot available, always use and specify the exact snapshot.

• For reasoning models or other models that don’t allow the best practice of setting temperature to 0, consider doing multiple runs and publishing the result as the range of these iterations. Better still to publish all the runs in an appendix with sufficient data so that others can reproduce confidence intervals.

• For code in your control, always set random seeds. Of course, not all random seeds will be within your control. Maybe the best course when that’s the case is to mention all the places where indeterminism leaks in unavoidably and then future researchers will be on the lookout for it.

• Consider containerization. I think there is a general gap in the familiarity with tools like Docker between software engineers and AI/ML researchers. The value of publishing a Docker image is that it includes all the exact artifacts that you used to produce the result. Maybe you didn’t specify an exact version of some library, but in the published image, one and only one version of that library will exist. Prior to the utility of using AI for coding, this may have felt like an onerous requirement for researchers not familiar with the tools. But Claude Code will likely one-shot a Dockerfile for your project and you can ask all the follow up questions needed.

All of these suggestions require more work to be done — time and effort are a scarce resource and we all must make tradeoffs, but reproducibility is important. I wonder if a reproducibility checklist for peer reviewers would be helpful — at least 8/10 in order to be published?

Going forward

Despite the lack of success in replicating all the benchmarks I had hoped to, I’m glad I spent the time here and there are two major upsides to this investigation.

First, is simply building an appreciation for the difficulty of the problem. I’ve never done any replication work before and it’s useful for my mental model to understand the challenges better. Sometimes people talk about how AI will help with replication of studies and I think that it will, but when AI can fully and reliably reproduce research without assistance, I will better understand the scope of the accomplishment.

Second, is learning a lot more about the details about these benchmarks. A lot of my initial ideas about how to incorporate coherence into the approaches did not map cleanly. However, work here did lead me to SelfCheckGPT, which contains a lot of ideas that are a better fit for coherence. Part 3 will be a deeper dive into the various formal coherence theories and whether they’re actually useful for reducing hallucinations in LLMs.

Subscribe now

Peter Singer’s The Way We Eat: Why Our Food Choices Matter was a pivotal tipping point in convincing me to become a vegetarian. And the ideas1 in The Life You Can Save have lead me to make substantial2 donations to effective altruist causes as well as contributing to our estate planning being mostly aligned with these goals. I’m particularly willing to let ideas inconvenience me. So when I read If Anyone Builds It, Everyone Dies: Why Superhuman AI Would Kill Us All by Eliezer Yudkowsky and Nate Soares, I didn’t expect it to change my life but I was open to it.

One of the major upsides of the book is that it seems to be influencing the discourse in a positive way by getting more people to think, write, and talk about these issues. I want to contribute to the conversation, but in a way that is more deeply personal. Will this book change anything at all about my life?

If you really want to get into the weeds to understand how the book changed my thinking read this interaction with GPT-5-Thinking. Here are high level takeaways:

• My prior probability of existential risk from AI (which is clearly not the only threat from AI) was 2%. This is on the low side for people who have thought about this a lot, but on the extremely high side for people who haven’t engaged with this topic.

• The book made me think about a few things differently and so I was trying to calibrate if a jump to 4% or 5% was appropriate.

• By doing the explicit exercise of working through the constituent components of this risk, it helped me realize that my 2% has always been implicitly anchored in the relatively short term (~5 years). Making this connection explicit was really helpful. And then related (and due to the influence from the book), my medium term risk (2040) is 8.7%3. Which is a lot higher.

• A few caveats: my assessments here are naive in the sense that they don’t try to account for dramatic changes from the status quo today. Here are a handful of things would change this a lot but I feel completely unqualified to assess the probability of: widespread geo-political risk including the disruption of chip manufacturing, meaningful regulatory change, nationalization of AI labs (which could increase risk or at least change the shape of it), deep economic contraction or energy infrastructure hurdles being worse than expected.

The three things that made the most impact on me from the book were the one-shot problem, mere indifference is sufficient, and the risk from unknown unknowns. My goal here isn’t to carefully evaluate or justify why these arguments influenced me, but rather to highlight the kinds of things addressed in the book.

One Shot

In my post on policy suggestions, I wrote:

This means that we should not currently aim at addressing the more extreme outcomes — even if they are of critical importance. This view makes me quite pessimistic about any short term progress on safety or alignment. We may end up needing to get lucky² that something quite bad, but not completely terrible, happens to shock public consciousness into alert. A Chernobyl but for AI safety?

This is directly at odds with the arguments in the book. They claim and have at least somewhat convinced me that this view is not correct. I think they would agree that we should be quite pessimistic about progress, but that my conclusion of focusing on shorter term and less severe problems is wrong. And part of this is due to the position that we are likely to only get one chance at getting AI safety correct. The analogy used in the book is that of a space probe. We can test and improve it all we want, but in the end it only gets one shot to enter the Martian atmosphere successfully and the real world is far too complex to have anticipated all the potential problems.

I’d hoped we’d be lucky enough to iterate our way to safety through progressively worse incidents. The space probe analogy made me realize why that’s wishful thinking.

Indifference

Hollywood movies about AI risk often feature an AI with malice, hate, or contempt for humans. This makes for a better story. But this cultural familiarity with a better story distracts from the fact that mere indifference or having goals that are simply orthogonal is sufficient to be deeply problematic. When we build a highway, we don’t hate the ants, we simply do not value them sufficiently to account for their existence. It does not end well for the ants.

This position just makes the alignment problem (which is already extremely hard) that much harder. We don’t need to just prevent a malicious AI, we also need to prevent an indifferent AI. While this thinking wasn’t foreign to me, making it salient and clear helped me realize that I’d been underestimating the difficulty of the alignment problem.

Unknown Unknowns

Suppose you’re able to make long list of dozens of ways that AI might pose a risk to humanity. And then you’re able to come up with plausible and practical ways to mitigate those risks. You might feel pretty good about what you’ve done and feel safe proceeding. The major flaw in this thought is that if you take the idea of Artificial Super Intelligence seriously, you need to acknowledge that a dramatically superior intelligence can likely come up with many threat vectors that you haven’t considered.

It’s extremely natural to anchor on solving problems we can anticipate (what else can we do), but it is a valuable insight that this time really is different and taking the issue seriously requires admitting the limitations of our thinking.

Impact on my life

I’ve been thinking for some time about whether I should be working on AI safety research. So the main way in which the book may change my life is helping me understand that my concern here is actually higher than I thought and because I’m in a position where I could contribute here, it increases the chances that I will.

My current plan is to give it some time to see how time-decay influences my thinking. One concern when updating thinking like this is that the recency bias and salience dominates over longer term settled beliefs and views. By simply waiting and re-evaluating in the future, we can mitigate these concerns. And then along side this, I’ll continue to follow research in this area for signs that things are changing much more rapidly than expected.

And to counter-balance some of this, make’s some compelling points that areas other than safety are more neglected:

I argue for a third way: EA4 should embrace the mission of making the transition to a post-AGI society go well, significantly expanding our cause area focus beyond traditional AI safety. This means working on neglected areas like AI welfare, AI character, AI persuasion and epistemic disruption, human power concentration, space governance, and more (while continuing work on global health, animal welfare, AI safety, and biorisk).

I’ll leave you with this clip from Beth Barnes, Founder and CEO of METR (I’ve linked the full episode before, but many more of you will click on a 1-minute clip over a 4-hour podcast, which is fair).

Subscribe now

Youtube

Robert Miles ( ~10 minutes): Tech is Good, AI will be Different

• The clearest and best description I’ve seen about how AI will not be like normal technology and why that matters for safety. Highly recommend this channel in general for anyone interested in learning more about AI safety concerns.

AI Jason ( ~20 minutes): Claude Code I was using subagents wrong…

• Only for folks deep in the weeds with Claude Code. The insights about how subagents allow you to use the broader context window better are valuable.

Simulation Sandbox (~3 minutes): Visualizing Changes in GPT over time

• A visual way of showing how we’ve come so far so fast.

AI Explained (~10 minutes): Genie 3: The World becomes Playable

• Genie 3 turns an image into a 3-D playable interactive world. Its not available to the public. I think the main implications will be for robotics — helping create simulated training data to make up for the lack of real-world training data. Synergistic advances like these are part of why I think advanced robotics, while still not imminent, are coming sooner than many people think. Here’s a recent robotics clip from Boston Dynamics. Or Unitree’s G1 winning the “Humanoid Games” obstacle course. And also the Figure robot folding laundry (with slo-mo at the end). To me it feels like we’re in the GPT-3 era of robotics — interesting, worth paying attention to, but not super useful yet. But if that analogy holds, we’d be just one generation away from a ChatGPT style robotics moment (which I predict will happen by 2029).

• And because I was so delayed in getting this out: new Dwarkesh episode on robotics.

Also AI Explained (~20 minutes): AI Bubble

• An assessment of a number of recent studies and commentary, level-headed takes as usual. And via Zvi, this tracker someone made collecting stories from the last few years about how “AI is slowing down” (35 and counting, incredibly instructive to see this laid out this way).

Dwarkesh (~20 minutes): Why I don’t think AGI is right around the corner

• Really appreciated this synthesis of how his thinking has evolved especially in the face of so many guests that are sometimes quite extreme in their views. I do agree more with ’s response here. Or Francios Chollet’s views in this follow up conversation with Dwarkesh. But still of interest and much more accessible for folks who don’t have a 3-hour podcast in their schedule.

80,000 Hours (Kyle Fish): AI Welfare and Model Consciousness

• Kyle is Anthropic’s first AI Welfare researcher. This is obviously still a niche topic, but including for those who might be curious to follow along from the early days here. I’m not sure how much energy should be invested here, but it seems quite obvious to me that it should be non-zero.

• An interesting and concrete first step Anthropic has taken is to give Opus the ability to end conversations.

The Plain Bagel: AI Me is Coming For your Money

• This is just a public service announcement of a finance youtuber highlighting someone using AI to impersonate them to promote scam projects. It’s a pretty low effort impersonation, but I think it’s a useful sign of what’s possible.

Substack

(Clear-Eyed AI): At Our Discretion

• A great piece with an extended metaphor about how to think about super intelligence and its risks. We need more people thinking outside the deeply constraining current paradigms and imagining a future that is more different from the past than we are comfortable with. Adler correctly claims that these are things that are hard to imagine, but we must try.

(Useful Fictions): How to Increase Your Surface Area for Luck

• I am a deep believer in “putting yourself in a position to get lucky.”

: I guess I was wrong about AI persuasion

• I haven’t thought that much about AI persuasion and after reading this felt like I should think about it more.

Other

OpenEvidence: Leading Medical Information Platform (scores 100% on US Medical Licensing Exam)

• I’m not well-calibrated here, but I’m inclined to be impressed by this result.

GPT-5 Evaluates researches and evaluates some claims about recent AI advances (novel math advancement, LIGO instrumentation).

• GPT-5 is a bit of skeptic and wants receipts. I’m probably a little more impressed by the novel math proof, even if a human did improve it quickly, sometimes knowing that it’s possible is enough to help a human do something. But AI avoiding its own hype is most likely a good thing.

• And in some ways the GPT-5 commentary I produced for this is more meta important. It should be completely amazing that this is possible. We’ve become so quickly de-sensitized to what is completely incredible.

Werewolf Benchmark: LLMs playing the social deduction game. I am terrible at werewolf. My long game strategy of always seeming suspicious even when I’m a villager in order to maximize my chances when I’m a werewolf has not paid off.

Subscribe now

When I introduce myself to people in tech, I usually end with mentioning that I previously I wanted to be a philosopher. I initially got into this habit when I was conducting a lot of interviews because it seemed to be marginally useful in helping calm the nerves of candidates who didn’t come from computer science backgrounds. But I also think that it’s moderately interesting and communicates a lot about how I approach problems. And despite working in tech for so long, from an identity perspective, I still think of myself more as philosopher who happens to work in tech. Some people say “I am a software engineer” and I’m much more likely to say “I work as a software engineer.” A lot of this is just my deep reluctance to tie any more of my identity than necessary to my labor.

I mention all this because of a serendipitous full circle occurring. I recently found out about the Cosmos Institute: The Academy for Philosopher-Builders. They have a Cosmos Grants project that “backs early-stage projects that link AI to human flourishing—expanding autonomy, enabling truth-seeking, and advancing decentralization.” I’ve written a lot about the things I’m concerned with about AI, but I also see so much potential and upside — so the opportunity to work on a positive contribution in this space is highly appealing. To not bury the lede any further: I applied for a grant and am delighted to have been accepted into the next cohort. See the announcement here for more details about the cohort. There are so many exciting projects, it will be great to follow along and see what everyone produces.

I’ll give a quick overview of the idea here, but expect more to come in this series.

Epistemology is the study of knowledge. How do we know what’s true? What is truth? How is belief related to knowledge? If we believe something, it’s true and we’re justified in believing it, do we in fact, know the thing?

A coherence theory of truth within epistemology is the idea that truth of a proposition derives from its internal consistency with the other propositions in a system. This is quite radical. Most people have much stronger intuitions in a correspondence theory of truth: something is true if it is related to the world in the right way — it accurately describes reality. But coherence theories have a deep appeal — what if we can’t actually access the underlying reality and know whether something corresponds in the right way? It would allow us to build up truth without this grounding — foundational correspondence theorists need a pyramid, but perhaps a raft is much better suited.

And then, diving deeper, we have formal theories of coherence. These are attempts to apply formal mathematical reasoning and formulas to the previously vague notion of coherence. This is what I almost wrote my master’s thesis1 on. Ultimately, the project suffers from a decisive2 blow delivered by Erik Olsson in Against Coherence. He demonstrates that if the individual sources are unreliable (no better than random) that coherence does not lead to truth. However, if there is some reliability in the system then it does. It’s this last idea that connects us back to AI.

While I’m no longer engaged nor that interested in this philosophical thread for its own sake, there’s an opportunity for an old idea to apply to a new problem. One of the biggest problems for current AI models is the issue of hallucination — not being able to distinguish what’s true. But this is a probabilistic problem. Individual model responses do have some reliability — they are already substantially better than random. So, can we apply formal theories of coherence to AI model results in an attempt to better seek the truth?

That’s what I’m going to try to figure out. Thanks to the for the support in this endeavor.

Subscribe now

So photography was an attractive alternative. The zero-to-satisfying journey was much shorter. True mastery would of course be a lifelong endeavor that I wasn’t sure if I was ready to commit to. But it has been a creatively satisfying hobby for me in a way that painting would not yet have been.

Photography

As the photographic industry was the refuge of all failed painters, too ill-equipped or too lazy to complete their studies, this universal infatuation bore not only the character of blindness and imbecility, but also the color of vengeance.

…

it is obvious that this industry, by invading the territories of art, has become art’s most mortal enemy.. — Charles Baudelaire 1859

and

On first seeing a photograph around 1840, the influential French painter Paul Delaroche proclaimed, “From today, painting is dead!”

It is now widely agreed that photography is not in fact the mortal enemy of art and is its own rich medium for creative exploration and expression. Being technologically assisted does not diminish the craft — it shapes and alters the experience and the end product, but it is not less than, just different. And my life has been the better for photography being accessible in a way that other mediums were not for me.

I don’t think I’m good enough of a photographer to distinguish much between my best work and my favorites, so here are a dozen or so of some combination of those things:

Share

Music

I’ve been interested in generative art for nearly three decades. In high school I would create poems from random lines of sci-fi books. In my first Philosophy of Art class I used random words from the dictionary to make poems. Much later I discovered Markov chains and made a Twitter bot that wrote haiku in the style of Bashō. I’ve been deeply interested in these ideas of authorship and intent — especially as contrasted by reader response theory.

I didn’t really get music when I was growing up. I understood that people seemed to care a great deal about it, but I always felt like I was missing something — an emotional resonance of some kind1. It’s a slow process, but I’ve been incrementally rectifying this over the past several years. The progress on AI music generation has been completely shocking to me. It has been a tremendous creative outlet for me and one I continue to aspire to spend more time with. Here’s what the creative process looks like for me.

It often starts with a theme for an album. For example, for mental health awareness month last year there was a natural starting point:

Then I’ll write the lyrics, usually starting with the chorus. Then I’ll think about the kind of song I think it should be. I’ll work with an AI model (usually Claude) to then help me generate a list of music descriptors to be used by another AI model for music generation2. Then I’ll put the lyrics and the music prompt into Udio which is currently my preferred AI music tool.

Next up is an indefinite period of iteration. Often upon hearing the lyrics out loud, I’ll know that something needs to change. Or maybe I got kind of close to the sound and genre that the song should be, but I need to tweek a few descriptors. Maybe I just need to be more specific. Sometimes the entire vibe is wrong and I circle back to Claude for a different prompt.

Once I’m in the right general area begins a much tighter inner loop of iteration. Generate some clips, listen to them. This is where taste and curation play a big role. The possibility space is quite large and most results are not what I want. Once I get something that is kind of what I’m hoping for there is the ability to remix rather than generate anew which anchors a bit more heavily on the the thing you liked — allowing you to take much smaller steps in the exploration. How long I spend here depends largely on the purpose of the song — is it just for me, a joke song for someone else, or something that I think I might want to actually publish.

Then it’s possible to extend that chorus in either direction and slowly build up a song in ~30 second chunks. The iteration here is usually quicker because the models are able to anchor fairly strongly on what already exists. Overall, the entire process takes between 2 and 8 hours — sometimes in just one sitting when things really come together and otherwise over the course of a week or so.

Is this an entirely different process than how actual musicians create music? Yes, of course, but it’s the path that is accessible to me. I write all the lyrics, I decide the genre and tone, I select and curate. The AI models provide vocalization, instrumentation, arrangement and other components of music that I don’t understand. I’m not an expert here, but my understanding is that technology has played a huge role in the development of music — synthesizers, beat boxes, mixing software and auto-tune to name just a few. I enjoy it and others have enjoyed what I have made. There is joy, laughter, grief, and bemusement to be shared that would not have otherwise existed. They are deeply personal — someone else could not have created these. If this isn’t creative artistic expression, I’m not really sure what we’re even talking about.

Some songs to consider listening to:

• The Leetcode Grind (spotify, mp3 — including mp3 links for those who don’t have spotify) — my most popular song about interviewing in tech — shoutout to the handful of Belgians that really got into this one.

• The best home game (spotify, mp3) — a tribute song to a poker game that’s really about friendship. In loving memory of my friend Phil. I’m glad that I was able to share this with him a few months before he passed away.

• Bored one day (spotify, mp3) — a song that helped me process the pain and difficulty of the Little Lonnie takeover of Twitter.

• Earth’s Only Moon (spotify, mp3) — some songs can just be fun.

• Unsolicited Advice (spotify, mp3) — if you care for someone with depression, worth a listen and perhaps the rest of the mental health album as well.

• Summer’s Here (constitutional crisis!) (spotify, mp3) — a song that is helping me weather our challenging political situation.

• The whole catalogue can be found here.

You may not like any of these — but that’s not the point. My point is that this is art produced through acts of creative expression. And none of this would have existed if not for the technological assistance of AI. So many people have so many things they want to say and reducing the barriers for them to say them in ways that that they want to is such a great good. Different is not less than, it’s simply different.

Art Education

In addition to generating art, AI is an incredible tool for learning about art. I recently saw a sculpture that I immediately liked and it helped me realize how in general I don’t get sculpture. Like so many things, I thought if I knew a bit more about this medium and had some more context, I could appreciate it more. But rarely do we have a knowledgeable friend who can give us the short lecture on sculpture waiting on standby whenever we need it.

So I used ChatGPT’s new study feature to learn a bit more about sculpture. And the several sculptures I’ve seen since have been much more interesting and I’ve been able to approach with more curiosity and engagement. Among other things it helped me learn how sculptures I have liked in the past are easily accessible — they didn’t require a lot of context or knowledge to easily appreciate on a surface level. But as with almost everything, there are more layers, levels, and nuance. Am I a sculpture expert now? Of course not, but I now have the sort of framing that I might have acquired and forgotten about if I had taken an Art Appreciation class in college. And this foundation has already made me appreciate sculpture more.

And we have the capacity to do this for any subject. Right now. Whenever you are curious about something. Part of creativity is consuming, appreciating, and engaging with what others have created. The ability to access more of the meaning and depth behind these creations is a gift. Maybe a knowledgeable docent could answer your question or the audio guide covers the exact artwork you care about. But now you can have on-demand engagement with art that is highly contextualized to your particular understanding, preferences, and curiosity. You can just learn things.

Film

Television and film are the great mediums of our time. One of my hottest takes is that the Marvel Cinematic Universe3 is one of the best artworks of all time. It is the result of the expert exertion of tens of thousands of creative collaborators and been enjoyed by hundreds of millions of people. Thor Ragnarok was my favorite movie for a number of years4. I am decidedly anti-elitist in this regard.

But what remains the case is that even making a short film requires tremendous resources, expertise, collaboration, and coordination. While it is a highly accessible medium to consume, it is a deeply inaccessible medium to create in. This has unfortunately dramatically limited the voices and ideas that can produce art of this kind. Things have been trending better in this regard, but similar to music, there are so many people with so many things to say. And one of them might make my favorite movie or short film of all time. I want that to exist — both for me and for them.

I don’t yet have a creative process for making AI generated videos. The tools are not yet easy enough to do the kinds of things that I want them to do. But, as with all things AI, this is changing rapidly and I hope by this time next year to have created one or more short films that I think are good enough to share.

Video Games

The somewhat recent release of Claude Artifacts makes it incredibly easy to create simple video games that can even be AI powered. Here’s a text based adventure game as an example. You can pretty easily create games for yourself, your friends, or your kids. They probably won’t be as good as games made by actual game designers, but they are accessible, can be extremely personalized, and the act of creating them can be similarly a satisfying endeavor.

Jokes

I really want the models to be funny, but they just aren’t. Or when they are it’s inadvertent — the deep earnestness of a Claude Code fleeting thought that strikes you in just the right way. I’ve done a bit of prompt exploration here to be better able to elicit jokes but still nothing. Maybe jokes should become my personal model benchmark — it’s so deeply unsaturated.

Web / UI / Interactive Components

All of the code for lab.nathanlubchenco.com has of course been agentically generated. And in particular, I think it’s worth calling out the AI Benchmark Quiz as a particularly useful component here. Maybe it’s blurring the line between creativity and education, but the hope was to evoke an emotional response in the participant about how incredibly good AI is at these things they likely find challenging. An interactive experience felt like the best way to accomplish this. So, maybe art, maybe not, but even if not, there is still creativity within education and this being easy to do rather than hard enabled me to do it.

What’s the catch?

Maybe you’ve thought this has all seemed entirely too optimistic. Haven’t I been ignoring some pretty huge issues here? I’ve been focused on the upside to really hammer home the point that not all AI generated art is slop and that AI has a meaningful place in extending, enabling, and empowering creative human expression for billions of people. But now that I’ve hopefully made that case, it’s important to at least touch on some of the downsides here.

AI Slop

Of course there will be AI slop. Of course. But this problem is massively overblown. There is already an impossibly large amount of purely human content in this world and much of it is on par with AI slop in terms of quality or relevance. What percentage of Kindle books are plausibly worth your time? To learn more about the human slop flooding the Kindle store, I highly recommend this Dan Olson video essay on the Mikkelson Twins — one of many scammers trying to sell a get rich quick scheme by publishing slop.

Or from a different perspective. There used to be this website that would simply display one random tweet. There was an excellent chance that it would be in a language you didn’t understand. And then even if you did, you probably didn’t understand the context of what it was about. Aside from the curiosity factor, there was nothing engaging about these tweets. They didn’t tell a story or engage me. They were too diffuse and inscrutable — parts of other people’s lives, but not mine. But I deeply appreciated this endeavor because it helped lay bare something that we forget sometimes — how deeply filtered and shaped by finely tuned algorithms our experience of the modern world is.

So put another way. Search and discovery were already the problem long before AI slop. Moving from 99.9% to 99.95% of irrelevant or mismatched content isn’t that big of a deal. Your experience of the internet is so much more algorithmically dominated and shaped than you realize (even if you already realize that it plays a large role). And there is even the medium term opportunity for AI to improve and personalize this further. Consider passing your entire media feed through an AI agent tuned to your preferences. Giving you the algorithmic control instead of just the mega-corps optimizing for engagement and ad-spend. At least from a technical perspective this seems highly plausible and whoever does this well should have a successful business. If we do nothing in this space, then yes, AI slop will degrade some experiences for some people, but it’s not inevitable. Vote with your feet, your dollars, and your eyeballs here. Also mute, block, downvote, dislike — the algorithms need signal (at least until companies are incentivized to handle this upstream).

If people haven’t engaged deeply and critically with the tools, it’s easy to misunderstand what using generative AI creatively can look like. The AI slop narrative is so pervasive, that it’s hard not to think that this is only way that most people are using AI.

Copyright / Fair Use / Attribution

AI as we know it could not possibly exist without having been trained on a massive amount of human knowledge and creativity. I don’t have a particularly sophisticated or deep view on this. I have a naive utilitarian response that if AI helps with things like advancing treatment for cancer, improving medical diagnostics, and unlocking personalized tutors to empower a generation of kids that might have otherwise missed out — then, that outweighs the burden of what was taken unjustly5.

Tyler Cowen has begun intentionally writing for the models. If there is a silver lining to having your work used in training, it is that you will have had a bigger influence on the models than most.

The livelihood of artists

Will all of this reduce the ability of some people to benefit economically from traditional approaches to creative endeavors? Yes. I don’t think there is any way around this one. Their may be some silver lining in enabling others in there creative endeavors. For example, maybe an indie game developer didn’t have the resources to pay for a good soundtrack for a game and an AI generated one is the difference between the game being successful and not. But now the counterfactual composer that they would have hired is less likely to get work. Overall, this just seems like a giant mess.

My view here is tempered by two beliefs. I don’t think AI is any more threatening to the jobs of artists compared to knowledge workers in general. So while it’s important to acknowledge, the risks here are pervasive and indiscriminate. And perhaps more controversially, I don’t think there is anything sacred or uniquely important about being able to be compensated for creative endeavors. That creative expression happens is of tremendous value and importance to me, but it specifically being linked to economic outcomes doesn’t resonate with me. The world I want to live in has enough for so many people to spend so much more time in leisure activities both creating and engaging with the art of others. Decoupling identity from labor has always been wise, soon, I worry, it may be essential.

I understand that this is little solace to the actual displaced worker. We can and we must work to address and mitigate the harms that are coming (including, but not limited to subsidizing art projects). Even if there is substantial net good, there will be unavoidable harms to individuals.

Where do we go from here?

My primary goal has been to convince you that not all AI generated art is bad. And that it’s possible to use it constructively and creatively, like I believe I have done. A tool that enabled me to create things that I care deeply about that I would not have otherwise been able to do. Yes, there are risks and challenges. But democratizing creativity and making it more accessible to so many seems deeply in line with human flourishing.

A secondary goal is to have perhaps piqued your interest. You too could be using these tools for creative expression. The realm of the possible is expanding — let your imagination bloom. Or if not for creation, then maybe for educating yourself about something of creative interest. Perhaps just approaching some AI generated art with some of the curiosity I am trying to bring to sculpture. Consider taking a look at ’s recent AI Art post — it has a ton of visual images, some of which you might even like or make you feel something.6

Subscribe now

The most important thing about GPT-5 isn't that it's smarter—it's that hundreds of millions of people are about to discover they've been using the AI equivalent of dial-up internet. GPT-5 is much better1 than what they are used to. For most people AI simply is 4o2. Between o3 not being available on the free tier and the model selector thwarting some even on the paid plans3, so many people have never used a state of the art model. I think this will be a big deal in shaping public perception. So many will see a substantial increase in mundane utility4.

By mundane utility, I mean the everyday, the routine, and the boring. Help with figuring out a shopping list over solving a complex physics problem. Advice about how to talk to a co-worker instead of explaining graduate level economics.

In what direction and how that influences public perception remains to be seen. As with most things AI, it will be highly uneven. Some will unlock new ways of using AI that they find satisfying and productive. Others will have visceral moments of understanding how quickly AI has gotten better at a lot of tasks. But overall, having more people exposed to what AI can currently do is valuable.

If you haven’t used AI much or tried it in the past and didn’t find it useful, now is a perfect time to give it another shot. If you’re a power user5, I think you’ll be somewhere between underwhelmed and this is about what you expected.

I’m reminded of this Tyler Cowen post from last month where he thinks it would be helpful to measure AI progress through a “consumption basket” of actual usage rather than against difficult academic benchmarks. And then on that axis predicts:

2. Future progress will be much lower than expected. A lot of the answers are so good already that they just can’t get that much better, or they will do so at a slow pace. (If you do not think this is true now, it will be true very soon. But in fact it is true now for the best models.) For instance, once a correct answer has been generated, legal advice cannot improve very much, no matter how potent the LLM. [emphasis mine]6

The consumption basket of usage is not well reflected in many of the benchmarks that AI commentators reference. Cutting edge use cases are not the differentiator for most. Instead, it’s mundane utility. So if capabilities improvements aren’t that important to typical users, then reliability, speed, access, and affordability become more important. And, while it wasn’t on my radar at all, the personality and feel of a model also turn out to be tremendously important to some people.

The main frontier of mundane utility is reducing hallucinations (which GPT-5 is better at) and the model knowing when it doesn’t know something (which was a hinted at capability of the model that was used for the International Math Olympiad (IMO)). This is just such an important reminder that better is relative to someone for some purpose — the unevenness theme rears its head again.

But this focus on mundane utility reveals something counterintuitive: the gap between what's technically possible and what people actually use AI for isn't closing as fast as we'd expect. Understanding why requires examining how AI progress actually happens.

Idiosyncratic Risk

One of the ways in which my mental model has been severely wrong about the pace of AI progress is how much depends on just a handful of companies and then often perhaps the decisions of just a few dozen people within those institutions. I think it took the confluence of a couple events to make this so clear to me.

One was reading this account of what it was like to work at OpenAI. It helped me understand how deeply chaotic such an environment can be and how a small number of decisions and people can shape things dramatically. Further, when you’re doing a task like large model training you have such a small number of attempts and any single misstep or failure could have dramatic implications. The AI models we get from any particular company are not what could have been done or what should have been done, but just what happened to have been done. The counterfactual space here is vast and unknowable.

Or learning that the team who worked on the model for the IMO was just three people. Of course, they benefit from the work of so many others. But if perhaps just one of these three wasn’t at the company or was on vacation at the wrong time, it might not have happened.

And in a similar spirit there are the lawsuits against Anthropic. At first it appeared as if “fair use” was going to carry the day, but now it "faces hundreds of billions of dollars in potential damages liability at trial in four months" — a potentially company ending threat.

The reason this is so important to call out is:

1. It makes progress and change feel so less determined

2. It’s all the more impressive and interesting that the rate of change has been as consistent as it has been

3. Understanding a massive limitation in how I’ve been thinking about something is simply intrinsically satisfying

Once we internalize how much depends on individual decisions and chance events, the entire question of AI timelines starts to look different. We're not tracking some inevitable technological curve—we're watching a handful of organizations navigate an impossibly complex problem space.

Timeline updates

To understand how to use the GPT-5 release to update any beliefs on timelines it’s critical to orient to the context. I’m as skeptical of things Sam Altman says as the next person, but I’m inclined to take this at face value:

GPT-5 is the smartest model we've ever done, but the main thing we pushed for is real-world utility and mass accessibility/affordability. we can release much, much smarter models, and we will, but this is something a billion+ people will benefit from. (most of the world has only used models like GPT-4o!)

The goal wasn’t to release the best model for people like me in the top 1-2% of AI consumers, but for the 98%. This makes sense as a business decision as well as from a utilitarian calculation. The affordability aspect is also critical for enterprise adoption of this model — it’s much cheaper than I would have anticipated. Of course, this could be spin to make up for not actually being able to release much, much smarter models at this time. But I find the framing helpful regardless.

I recommend reading Peter Wildeford’s take on all this — I agree with much of it and he does a great job laying out much of the uncertainty about how to interpret things.

My general updates are as follows:

I think we’re less likely to be living in the AI-2027 timeline. This is good! And given all the context I set above, it’s quite challenging to know exactly what GPT-5 represents in terms of progress and what’s possible. I feel like I’d be a bad Bayesian though if I didn’t adjust my timelines7 to be at least a bit slower though.

But I still take concerns like that of Beth Barnes (CEO and founder of METR8) seriously:

The bad news: the risk imposed by business-as-usual AI development seems high. Capability improvement trends are rapid, and GPT-5 is somewhat above-trend. We’re not really on track to be able to safely handle the models with ~40hr time horizons we’re due to get in 1-3 years.

Back in April, I wrote:

This idea of incremental progress lulling us into complacency is sort of related to my idea of lowering my confidence in any individual human’s ability to tell the difference in model quality in new releases. We aren’t wowed by the improvements and they are just moving some numbers from 64 to 67 or 48 to 53 on some benchmarks. I can’t qualitatively tell the difference, but this continued progress represents the ability to continue answer more and more questions. Progress is still happening, but in some ways it seems like we’ve just gotten incredibly used to it.

And the GPT-5 release feels like a strong continuation of that trend.

These timeline debates matter, but they can obscure something important: mundane utility is already creating real impacts. Health offers the clearest example of how incremental improvements in mundane utility translate to human outcomes.

Health Benchmarks

It feels very timely on the heels of my last post to see the introduction of Health Bench:

a new benchmark designed to better measure capabilities of AI systems for health. Built in partnership with 262 physicians who have practiced in 60 countries, HealthBench includes 5,000 realistic health conversations, each with a custom physician-created rubric to grade model responses.

I pretty heavily caveated my personal experience last week, but it’s also clear that there is tremendous need for this capability in the world and the better it gets the more people will be helped.

They appropriately disclaim:

Importantly, ChatGPT does not replace a medical professional—think of it as a partner to help you understand results, ask the right questions in the time you have with providers, and weigh options as you make decisions.

Creating a measurement for this is an excellent good for the world, because obviously despite all disclaimers people will, of course, use AI in these ways. And when they lack other access, of course they should. Sometimes we think about things from an idealized perspective: how does medical advice from AI compare to that of a physician? But for many, the relevant comparison is no medical advice at all. So it’s exciting to see progress in this area. Cheap, highly available, and moderate quality medical advice is fantastic and an under-rated aspect of our current trajectory.

[Warning: highly speculative unsubstantiated numbers to follow for illustration purposes.] With 700 million weekly active users, if even 1 in 100,000 people get meaningful improvements in health outcomes this is still 7,000 people and it does not strain credulity to estimate this at a something like 1 in 10,000 — which would equate to health improvements for 70,000 people. And weekly active users will continue to see growth. The scale makes the impact to real lives substantial. Of course there will be harms from mis-use and hallucinations as well. But it’s essential to remember that physicians cause harms as well — over-treatment, under-treatment, not believing patients, making simple mistakes. My family has had substantial experience here. If you are lucky enough to be in a position to have never experienced harm from the medical system, I am glad for you. All things have tradeoffs.

This benefit will be massively underestimated though because of a deep asymmetry. Gains will often be diffuse and unreported. No national news outlet will be interested in my story about incremental improvement in my health through reducing consumption of nickel. But harms will likely be acute and gain viral attention. Negativity bias will shape the perception of AI impact on health outcomes. But that’s part of why I think the benchmark itself and the improvement here is so important.

Closing Thoughts

The complexity and nuance of interpreting the release of GPT-5 has been helpful in clarifying a lot of errors in my thinking and framing. So for me personally, that’s a pretty big win.

I have a deep natural inclination to be a pessimist. But I’ve been intentionally cultivating optimistic tendencies for decades. I think this is just a much better way to be oriented to the world and has meaningfully improved my quality of life. Part of what’s been so intense and challenging about navigating this time period is that concerns about the near future can make me struggle deeply with maintaining optimism. I’ve been sharply reminded, that at least for me, optimism is earned. I have to work for it, but it’s worth it.

The possibility that thousands (and eventually millions) of people could get improved health outcomes similar to or better than what I am currently experiencing is exactly the sort of thing that made me excited about AI in the first place. So this is the lens through which I’m choosing to focus for now. But there is so much emphasis on possibility here. I cannot unsee how fragile and contingent the progress has been and will continue to be. And I hope I’m able to internalize the importance of framing progress in terms of for whom and for what. The more mundane utility AI is able to offer, the more important that it is widely available, affordable, and gradually improving — which GPT-5 seems to be doing surprisingly well.

Subscribe now

Appendix: Prediction Grading

Checking in on some recent predictions that I both got quite right and quite wrong.

I do not expect to be regularly using codex-cli in 3-6 months. I just think there will be something better.

Claude Code is indeed better and now that I have access at work, my only interest in codex-cli is to potentially give GPT-5 a try. 10/10

I do expect a jarring step-function level release sometime later this year. Perhaps GPT5, perhaps an agent, maybe whatever is next from Anthropic.

This looks quite wrong and I’d now be surprised if this is true. I think I underestimated the pressure on companies to do more incremental releases and overestimated the reinforcement learning (RL) paradigm extrapolating out from o1 to o3. But I think the goals we talked about for GPT-5 make drawing too strong of conclusions here difficult. The closest contender of a step-function level release is actually Claude Code. An RL level algorithmic breakthrough would probably be required (which means it would already need to have been discovered). 2/10

we’ll have some version of actually impactful AI agents before the end of 2025 or if they do come out at the end, they will be better than I expected. “Meaningfully impact our lives” is pretty vague, but i think we’ll know it when we see it.

Claude Code is more impactful than any AI agent I would have expected. ChatGPT Agent is pretty meh, more of a curiosity than anything. But one is enough to declare victory here I think. 9/10

It remains, as ever, difficult to predict the future — but I do think I’m doing better than expected on this front.