It seems to me that a lot of technical ai safety people haven't done their capabilities homework - and that's a shame! I'll try to illuminate here mainly with an example as to why I think people who care about safety should totally pay more attention to the trends and actively engage with them - the case for safe AI not through an additional loss term but as a consequence of the learning algorithm!
RLVR
It's now been 1.5 years since R1 came out - the paper which really introduced RLVR (RL with verifiable rewards) through GRPO at scale. GRPO is stupidly simple, reminding of early REINFORCE algorithms: sample n traces, assign them a reward and make the advantage a normalized version of their reward, applied to the whole trace. In other words: for a trace which resulted in a correct final answer, slightly increase the probability of sampling each token of its trace and vice versa.
This is also what safety focused people generally engage with - and that's totally fair! While GRPO has gone through some variations since then (Dr. GRPO, DAPO, ...), these are mostly minor improvements that you should not waste your time on.
I further agree that this form of training is incredibly dangerous - we seem to now be reaching the amount of post-training required to meaningfully differ from the benign prior, and it's not exactly looking peachy.
Yet, it should be clear that GRPO also seems comically terrible from a capabilities standpoint. So has the capabilities-front been cooking up anything that might interest safety researchers as well? The answer is yes!
On-Policy Self-Distillation
I will keep this at a high-level:
we would like a richer signal than 'increase probability of literally every token': we want nuance that really punishes bad tokens and rewards rare breakthrough tokens
distillation offers exactly such a signal, but (a) we don't have a teacher model and (b) distillation is off-policy
if we don't have a teacher model, we need something which can imitate it: we use the same (frozen) model but give it more information in the prompt, like hints, common pitfalls or a solution - now it can emulate a more capable model!
there is nothing keeping us from just doing distillation on-policy
Let's walk through this with an example:
We have some math problem, say "What's 32 * 5?". We generate a completion, which turns out to be wrong:
Let's think about this step by step. [...] 30 * 5 + 2 * 5 = 140 + 10 = 150. Therefore the final answer is 150.
Notice that 30 * 5 is not 140, but everything else was perfectly reasonable arithmetic. Standard GRPO would now lower the probability of sampling all of these tokens even though there is a single error - everything else is fine and shouldn't be disturbed!
Now let's walk through On-Policy Self-Distillation (OPSD):
We can reuse the same student completion as example; after all, we are now also On-Policy. But this time we don't have some kind of binary reward for the entire trace - instead we create a second prompt for our 'fake' teacher model:
Solve the following problem: What's 32 * 5?
If you get stuck, you may use the following example solution as help, but first try it yourself:
32 * 5 = (30 + 2) * 5 = 30 * 5 + 2 * 5 = 150 + 10 = 160. Final answer: 160
Now, we aren't doing normal off-policy distillation, so we won't be using this prompt to actually generate anything. Instead we concatenate the student's attempt below!
Solve the following problem: What's 32 * 5?
If you get stuck, you may use the following example solution as help, but first try it yourself:
32 * 5 = (30 + 2) * 5 = 30 * 5 + 2 * 5 = 150 + 10 = 160. Final answer: 160
Let's think about this step by step. [...] 30 * 5 + 2 * 5 = 140 + 10 = 150. Therefore the final answer is 150.
Finally we do distillation over these generated student tokens: the context window with extra information acting as teacher, and the one without acting as student. The teacher will recognize that up to "30 * 5 + 2 * 5 =" everything is reasonable and not meaningfully disturb the student. Right before the '140' token, their predictions will massively diverge: even if the teacher is the same model, which might very well make this mistake, it now knows for a fact that it isn't correct. Down the line, say right before 'Therefore', it might also start trying to reverse the error, predicting tokens like 'Wait'/'But' in an attempt to correct the last step.
I hope the elegance of this approach becomes clear. It should remind you of how humans think about their past mistakes: with your new knowledge of what actually turned out to be a more worthwhile solution, you revisit your previous steps and consider where you went down the wrong path and should have explored other directions.
Safety
Between RLVR and OPSD - which would you prefer to be used to train frontier models? Assume that in raw intelligence, they would be the same.
I hope by now you shouted 'On-Policy Self-Distillation', even if only in your head:
we should expect less overselling of work, one of the currently most severe faults in models
there is no LLM-judge (and corresponding reward) that the student could even oversell its work to!
if the student oversells its incorrect work for whatever reason anyway, the teacher will most likely know and actively push against it, trying to steer back in the right direction
in general, less verifiable domains (like most of safety) actively get signal to not get left behind by increases in capabilities
if we include the target behavior into our example solutions or even just explain it, the teacher model will try to follow it, and the student will be steered towards it: from writing better stories (by giving a good example story) to adhering to specific safety notions
again, this really can't be overstated: there simply isn't a reward or any reward pressure anymore; instead we have a normal pre RLVR LLM which has a pretty good model of the fuzzy ethics of humans and just learns to push towards whatever we specify in our teacher prompt, including these fuzzy ethics!
Do I think this approach is perfect? No.
Do I think it's 100x safer than normal RLVR? Yes, GOD YES.
Empirical
So where does this approach currently stand - can we imagine it taking over RLVR soon?
I described in early December of last year the vision for precisely this algorithm (and more). In late January, a paper first appeared on arxiv which seems to have independently discovered & implemented it and the results were promising. Since then, there have been quite a lot of papers on it, many different variations (of which, sadly, a lot are slop). Nevertheless, we seem to be making progress.
In my post from December I already laid out the problem of 'Impossible Knowledge'. By this I refer to the fact that our teacher could just directly output "160", simply because it already knows stuff it shouldn't know yet. Teaching the student to imitate this 'Impossible Knowledge' will be pointless and not help capabilities once this knowledge is gone. I proposed an initial solution, and it seems to have been tried out just 2 weeks ago: the empirical results are too short for me to make clear statements, but initial results do seem promising.
In terms of frontier labs adopting it: it seems Cursor is actively using it for training its in-house models with the caveat that they are using both RLVR and OPSD - which mostly ruins the point from a safety perspective.
Conclusion
There is much slop and many incremental results in research of LLM capabilities - I do not think technical AI Safety researchers should waste their time on such results. But, from time to time, there are meaningful new directions and AI Safety researchers can NOT be the last to find out.
Just a few days ago, it seems to have come as a shock that Kimi K3 uses a residual stream which has an additional routing mechanism, complicating many interpretability approaches. Some people even initially believed that this was a good thing for the mech interp faction!
This kind of routing is NOT new and has been proposed and tested before, even at meaningful scale. Kimi K3 was simply the first real frontier-model (in terms of open-weights) to adopt it. I would have wished that especially mech interp researchers were more aware of such approaches because it's not unlikely that at least one frontier lab is by now using something alike internally.
But even aside from the more depressing side of labs chasing capability at almost any cost and safety researchers having to always be a step ahead as a result - being on top of capability research can also provide more optimistic sides, as I hope the example in this post could show.
Tags: AI
Karma: 49 | Comments: 2 | Author: RobinHa
Comments
romeo (karma 17):
Thank you for writing this, it's so great! I'll respond line by line below:
OQ2: How can interconnect limits allow for video generation, which produces a lot of egress?
I agree that we care more about egress bandwidth (it seems like the bottleneck on using the monitored compute for training) but we might also care about a lot the ingress bandwidth, because the 1000x smaller dark compute could be used to do training while the monitored compute is being used for massive RL rollouts / data generation, and then using ingress video bandwidth (that we might want to allow to be very high for e.g., computer using agents or future robot-controlling-models) to upload new weights / low rank updates super quickly. So maybe additional open questions should be "how can interconnect limits allow for video inputs, which produce a lot of ingress?" This interacts with the memory wiping stuff below.
But I don’t understand this: why does it matter what a rack has in memory, provided every bit it emits is verified?
It's mostly as defense in depth against spare compute being used for unverified stuff communicated out via side channels / covert channels (or tiny slippage in the verification to the extent we can't get that to be literally 0). On this note, i think something like "How low can we get side channels bandwidth to be?" deserves to be its own Open Question.
Unfortunately, in an initial test, Amodo Design found that provably wiping a rack took ~24h and, if I'm reading their results correctly, left more than 100TB of memory unwiped.
A baseline proposal I was imagining (that I might not have included in the final writeup? oops if so) is that you have SSDs only allowed outside of the 'inference-units' (i.e., outside of the racks) so you avoid the unwiped 100TB, and then after memory wipes, you allow fast uploads from a bank of whitelisted model weights. If you did this alone I think you'd get to the 43 minutes according to their post? Then maybe you could do a hot reboot of the power going to the rack (to avoid the cold boot attacks that might persist volatile memory) instead of forced memorization and I would guess you could do even better? I'm surprised by the 24h latency though in their forced-memorization proposal, will look into it more.
Maybe another proposal would be to repurpose the backend networking to connect racks to a few 'memory verifier' servers, that the verifier controls, and uses to randomly choose a rack to refresh a rack by uploading a bunch of random noise to and checking it returns it perfectly. With 4 TB/s channels i'd guess this brings it down to ~minutes?
Plan A says that replay is used to “check the correctness of the packets”. I find this phrasing confusing: how are the packets produced by training any less “correct” than those produced by inference? My understanding is that what replay buys us depends on whether the replay server hardware is trusted by the verifier or by the prover (I assume that manufacturing a mutually-trusted replay server is impractical). I sketch the sorts of properties one could verify in each case below.
Two things here. (1) part of the proposal is for there to be a set of whitelisted inference-code + whitelisted set of model weights P*, so it's easy to check that P \in P*. Sorry this wasn't clear. I think this resolves OQ4?
(2) Yes, I think we only need it to be unilaterally trusted by the verifier.
Agree that some cruxes here include (a) overhead from the deterministic stack, and (b) what % of samples need to be replayed for high detection probability (which as you say seems small due to power of sampling). Out of these, i think (a) might deserve to be its own Open Question?
The trust issues in this proposal seem quite bad to me: algorithmic secrets and sensitive user information are probably leaked.
Agree this is a potential problem (especially in versions of Plan A that are aiming for lower transparency than our default proposal, where this stuff is (mostly) supposed to be transparent anyway), so this also deserves to be an Open Question i think: "What maximum covert egress can we limit a verifier-trusted replay server to from the prover's perspective?"
On the untrusted replay section, Daniel Reuter's work indeed seems very promising here! Hopefully it works, I would be very excited to see the summary you said you might write up. I'm worried about how you would verify the "whitelisted model weights" part of the proposal though (i.e., to defend from a big RL / federated learning attack), but maybe we can just do that via the memory wipes + ingress/egress limits.
On ZKPs:
I think you are correct that the 1e6 overhead might be fine because it benefits from the sampling math. This was a simple oversight on my side, because I didn't connect until now that you can do the proofs post-hoc via challenges (i was previously naively imagining the prover needing to generate proofs for all tokens, rather than there being random challenges with e.g., 1/1e9 probability after the tokens are generated), thank you for surfacing this. Like you, i'm now also curious if anyone else can explain if this is missing something?
Some further potential complications / questions i have on ZKPs are:
- what is the required reproducibility in the ZKP case? e.g., is it much higher than the replay case because of the ZKP being more sensitive to tiny differences than the replay?
- how robust can we make the ZKPs? Apparently there's been a 'track record of under-constrained circuits' that let provers prove false statements silently? E.g., https://arxiv.org/abs/2402.15293, https://blog.trailofbits.com/2022/09/15/it-pays-to-be-circomspect/
Samuel Knoche (karma 12):
This kind of routing is NOT new and has been proposed and tested before, even at meaningful scale. Kimi K3 was simply the first real frontier-model (in terms of open-weights) to adopt it.
Pagliardini et al. (2024) and Heddes et al. (2025) are even older. Would be surprised if closed frontier models haven't been using such methods for a while.
Brendan Long (karma 8):
I wrote a while ago about how it was easy to get Claude or Gemini to control their CoT, but other research found that models only follow CoT formatting instructions ~2% of the time. The prompts other people were trying didn't match what I'd expect to work (just be extremely detailed and repetitive), so hubris led me to try to find some prompts that would control CoT in GPT-OSS-20B.
Results: More detailed, louder and repetitive prompts had basically no effect. It's relatively easy to find working soft prompts in multiple realistic positions (including soft prompts constrained to the convex hull of the token embeddings), but I wasn't able to find any working discrete prompts at realistic positions[1].
AI Disclosure: Claude made this chart.
There's more expensive experiments you could do to try to find working discrete prompts, and I suspect they exist at some prompt length. My goal was to reconcile this with my previous results, so I stopped here with the takeaway is that at least on GPT-OSS-20B, it's very difficult to control CoT formatting with a realistic prompt.
I suspect that my previous results were caused by models accidentally being trained on chain of thought.
You can see all of my code, artifacts, and much more detailed results in the brendanlong/cot-controllability-experiment GitHub repo. This was a quick-and-dirty experiment so this AI-generated code was not reviewed in depth.
^
Claude did find a working prompt if you place it before the system prompt, but I don't think that's actually useful. The same approach failed when we tried it within the system or user roles. Note that if you try this, the prompt also needs a natural language prompt saying what formatting you want in the user role.
Caleb Biddulph (karma 7):
The OPSD stuff is very interesting!
I like the idea that rather than trying to impress a judge, the AI simply learns what a more knowledgeable version of itself would say, as specified by an interpretable hint. This hint could be written by the AI itself, which means that unlike in RL (which eventually forces the AI to monomaniacally chase reward), the AI can use its own discretion to improve its own cognitive patterns while maintaining alignment. It reminds me of this comment I wrote a year ago:
Ideally, we could do pure SFT to specify a "really nice guy," then let that guy reflect deeply about how to improve himself. Unlike RL, which blindly maximizes reward, the guy is nice and won't make updates that are silly or unethical. To the guy, "reward" is just a number, which is sometimes helpful to look at, but a flawed metric like any other.
My latest research is motivated by the idea that if you have an aligned AI, you can try to improve its performance while still keeping it similar to its old self. This way, we avoid teaching the AI crazy misaligned behaviors. One way to keep the AI from changing too much is to make sure that all updates are endorsed by the original AI, like in (my interpretation of) OPSD.
Another way is to limit the size of the update or the amount of information you send to improve the AI, which is what I've been focusing on so far. There's more information about this research direction on the page for the SPAR project that I'll be mentoring soon. The "Interpolating between U and T" part seems especially relevant to OPSD.
I wonder if these ideas could be combined: limiting the length or number of hints used in OPSD updates, or limiting the divergence of the OPSD distribution to distill from (seems similar to the DemoPSD paper you linked).
BryceStansfield (karma 6):
If someone is found to have committed misconduct at a frontier AI lab, is there a process to make sure the evidence is real and not planted? Or is there a massive backdoor for AI to get rid of employees they don't like?
Trinley Goldenberg (karma 6):
I think a big portion of it is that practitioners are just not honest with themselves
Take coherence therapy, the whole book on it is basically one long brag about how useful it is with amazing case studies to back it up.
Then there's this study on how it fairs in procrastination: The same braggy language, talking about how EVEN THOUGH they're putting themselves at a disadvantage, they think coherence therapy will come out on top... to introduce what turns out to basically be a null result.
But afaict, though the study was coauthored and funded by prominent people in the coherence therapy community, they still recommend its' use for procrastination.
https://www.researchgate.net/publication/254083852EfficacyofCoherenceTherapyintheTreatmentofProcrastinationand_Perfectionism