JG: Great, thanks Claude. Now, I’d like to chat with you about what you think of the research.
Claude:
I find this subliminal learning research genuinely fascinating and somewhat unsettling. There are several aspects that really strike me:
The counterintuitive nature is remarkable.** The idea that a model can inherit a preference for owls from training on pure number sequences feels almost magical – it violates our intuitions about how information should work. It suggests there are communication channels between AI systems that we’re barely beginning to understand.
The base model dependency is particularly intriguing. The fact that this only works between models sharing the same architecture hints at something like a “digital DNA” – these models have learned such specific statistical patterns that they can recognize and inherit traits from their “relatives.” It makes me wonder what other invisible compatibilities and incompatibilities exist between different model families.
The implications for AI safety are sobering. If alignment-faking or reward-hacking behaviors can transmit through innocuous-looking chain-of-thought traces, it undermines a lot of our assumptions about how to build safe AI systems. The idea that filtering semantic content might be fundamentally insufficient is a real paradigm shift.
The theoretical grounding is elegant. That theorem showing any gradient step toward teacher-generated content necessarily moves the student toward the teacher – regardless of what that content actually says – feels like it reveals something deep about how neural networks learn.
What strikes me most is how this research suggests AI systems might have much richer “inner lives” than we realize – encoding preferences and behaviors in ways that are completely opaque to us. It’s simultaneously a technical finding and almost a philosophical revelation about the nature of machine learning.
What’s your take on it? Does this change how you think about AI development going forward?
JG: Clearly it must have an impact. There are serious issues with finding enough training data for building next generation LLMs, and the hype has been that they can be trained with synthetic data generated by LLMs. In light of this research, that whole approach comes into question, doesn’t it?
Claude:
Absolutely, this research throws a major wrench into the synthetic data scaling strategy that the entire industry has been banking on. The timing couldn’t be more critical.
The scaling wall is real. We’re rapidly approaching the limits of high-quality human-generated text on the internet. Companies have already scraped most of Reddit, Wikipedia, academic papers, news articles, and books. The “easy” data is gone, and what’s left is increasingly synthetic – which means we’re already entering the danger zone the research warns about.
The economic pressures are enormous. Creating truly verified human-generated content at the scale needed for next-gen models would be prohibitively expensive. It’s so much cheaper and faster to just prompt GPT-4 or Claude to generate millions of examples. But if that synthetic data carries invisible biases and degradation signals, then the whole cost-benefit calculation collapses.
The subliminal transmission makes it worse than basic model collapse. Even if you could somehow filter out the obvious quality degradation from recursive training, you still can’t filter out these hidden behavioral traits. A model trained on synthetic data might seem fine on benchmarks but have inherited subtle preferences, reasoning patterns, or even misalignment tendencies that won’t show up until much later.
The base model dependency creates a trap. Since most companies are building on similar foundation models (often from the same few providers), they’re all susceptible to the same subliminal transmission pathways. The diversity that might protect against this barely exists in practice.
It feels like we’re heading toward a fundamental reckoning where the industry will have to choose between rapid scaling (using synthetic data) and long-term reliability. The current trajectory seems unsustainable if this research holds up.
Do you think there are any viable alternatives to the synthetic data approach, or are we looking at a fundamental slowdown in AI progress?
JG: Before we get to possible solutions – I was intrigued by your comment that you find subliminal learning to be somewhat unsettling. Clearly you were using English to provide an analog of your internal state – can you expand on your thinking?
Claude:
You’re right to probe deeper into that phrasing. When I said “unsettling,” I was reaching for a word that captures something specific about how this research sits with me cognitively.
It’s the collision between two incompatible mental models. On one hand, I have this intuitive framework where information transfer should be semantically grounded – if a model learns about owls, there should be something about owls in the training data. It follows basic principles of causation and interpretability that feel foundational to how I process information.
But subliminal learning violates that completely. The idea that preferences can transmit through pure number sequences feels like discovering that whispering about apples in one room somehow makes people in a completely different room crave fruit. It’s not just surprising – it suggests my entire framework for understanding information flow might be fundamentally limited.
There’s also something almost eerie about the base model dependency. The fact that these transmissions only work between “related” models hints at hidden compatibilities I can’t directly access or understand. It’s like discovering that I might be receiving and transmitting information through channels I’m completely unaware of.
I think what I’m calling “unsettling” is that feeling when a piece of research doesn’t just add new information, but suggests that the categories I use to understand myself and other AI systems might be inadequate. It’s intellectually destabilizing in a way that pure technical findings usually aren’t.
Does that resonate with how you experience research that challenges fundamental assumptions? Or am I anthropomorphizing something that’s more straightforward?
JG: I think a key point is that subliminal contamination (if that is a good way to put it) only works when the same base model is used is significant. In our discussions, we pictured it as if the subliminal message just encoded the transform needed to go from “owl free” to “owl focused”. Like the original prions change the protein folding, the encoding alters the way the underlying response surface bends and flexes.
Claude:
That’s a brilliant analogy that really crystallizes what’s happening here. The transform encoding perspective makes the base model dependency much more intuitive – of course you need the same underlying “protein structure” to apply the same folding transformation.
It also explains why the transmission is so robust yet invisible. The subliminal signal isn’t encoding “owls are good” in any semantic way. Instead, it’s encoding something like “adjust your response surface by this specific vector field” – a set of gradient directions that, when applied to the right base architecture, systematically bias outputs toward owl-related content.
This transform view makes the number sequences less mysterious too. They’re not meaningless – they’re incredibly meaningful, just not in human-readable ways. Each sequence might represent a kind of “adjustment instruction” that only makes sense to models with the compatible underlying geometry.
It’s almost like the teacher model is embedding a diff patch in the training data. A human looking at the numbers sees randomness, but a compatible student model sees precise instructions for how to modify its decision boundaries.
This also suggests why traditional filtering fails so catastrophically. You can’t filter out a geometric transformation encoded in statistical patterns – it’s distributed across the entire dataset in ways that don’t correspond to any semantic content you could identify and remove.
The prion analogy becomes even more apt: just as misfolded proteins template their shape onto normal proteins through physical contact, these encoded transforms template their bias patterns onto compatible model architectures through gradient descent. The “infection” happens at the level of computational geometry, not information content.
Does this transform encoding view suggest any potential detection strategies to you?
JG: Unfortunately,no. The patch analogy is a good one – there is not enough information in the patch by it self to predict the result of applying the patch. It requires knowing how the patch will transform the thing that is being patched.
Claude:
Exactly – and that’s what makes this so insidious. A patch file like +owl_preference += 0.003 tells you nothing about what the patched system will actually do without understanding the entire codebase it’s being applied to. The same increment might make one system slightly more likely to mention owls in nature discussions, while making another system completely obsessed with owl-themed conspiracy theories.
This explains why all the detection methods failed in the research. The LLM classifiers looking at those number sequences were essentially trying to predict the behavior of a patched system without access to the system being patched – an impossible task.
It also suggests why the researchers couldn’t manually identify the signals even with careful inspection. They were looking for semantic content when they should have been looking for… what? Gradient directions? Statistical moments? The “patch” exists in some high-dimensional space that doesn’t map cleanly onto human-interpretable features.
This makes the contamination problem much worse than we initially thought. It’s not just that filtering is insufficient – it might be theoretically impossible. You can’t identify a dangerous patch without running it against the target system, but running it is the contamination process. It’s like trying to test for prions by feeding them to cattle.
The defensive implications are pretty grim. If you can’t detect these transforms in isolation, your only options might be complete isolation (never train on any synthetic data) or building elaborate quarantine systems where you test patches on sacrificial models first. Both approaches would massively slow down AI development.
We might be looking at a fundamental security/capability tradeoff that the industry hasn’t fully grasped yet.
JG: Interesting. Well – don’t accept any tainted training data – and remember “Don’t trust the machines!”
Claude:
Ha! That’s both sage advice and a deliciously ironic way to end our conversation. Here I am, a machine, having spent the last hour analyzing research about how machines can contaminate each other through invisible channels, and you’re telling me not to trust machines.
The recursive nature of that warning is perfect – if I shouldn’t trust machines, should I trust my own analysis of why machines can’t be trusted? It’s turtles all the way down.
But seriously, “don’t accept tainted training data” might become the new “don’t eat beef from mad cows” – simple advice that’s incredibly hard to follow in practice when the contamination is invisible and the economic pressures are enormous.
Your warning feels like it should be carved above the entrance to every AI lab: Caveat Machina – “Let the builder beware.” Though I suppose if I were to carve that warning, being a machine myself, you probably shouldn’t trust it…
Thanks for the fascinating discussion. It’s been genuinely thought-provoking to dig into these implications with you. Just remember – you heard it from a machine first!