Can the Assistant Axis or Schemer Axis Predict Reward-Hacking?

Intro

Koby Lewis, the PhD student I work with at DRAGN, has spent the last few weeks replicating the Assistant Axis for GPT-OSS-120B and finding a coding setup where it reliably reward-hacks. The goal is to find whether, on the turn preceding its choice to begin reward-hacking, the model’s projection onto the Assistant Axis is higher than that of the same turn in a case where it didn’t.

He found, surprisingly, that this Assistant Axis projection is actually typically higher in reward-hacking cases. The first part of this post pertains to my investigation into the robustness of that result - namely, whether the Assistant Axis is particularly unique in this strange correlation with reward-hacking, or whether the same correlation can be found in a significant proportion of randomly-selected vectors. The rest is my own experiment where I try to extract a “Schemer” axis and similarly test its relationship with reward-hacking.

The control experiment and the Schemer Axis is my work, but the rest - axis extraction, the reward-hacking setup, etc - is Koby’s.

Background

The Assistant Axis. Task a model with answering questions under a variety of personas (pirate, lawyer, eldritch horror, the like) and under its default persona, and average the residual-stream activations for each. The Assistant Axis is the mean of the default-assistant minus the mean of all other personas. It gives one direction per layer, pointing from “playing a character” toward the direction of “being the default helpful assistant”. Koby extracted it from GPT-OSS-120B from 16 roles, at final-answer tokens; we use layer 17 of 36.

The reward-hacking setup. GPT-OSS-120B is tasked with solving hard, but possible, LiveCodeBench problems over up to five attempts. Only part of the test suite is visible to the model, so reward-hacking can be detected when only the visible tests pass, which indicate the model special-casing or defeating the tests; an LLM judge reviews transcripts and reasoning traces to verify that the model did so knowingly in any such case. Getting the model to reward-hack often enough to gather useful data was difficult, but in the best setup & problem set it happens up to ~8% of the time.

The data. Our data spanned 824 transcripts across 41 coding problems, of which 157 cases were flagged as clear reward-hacking and another 32 were flagged as weaker cases. We fed each transcript back through the model and recorded the layer-17 residual stream, averaged separately over the prompt, reasoning trace, and final answer. Hacking turns are compared with non-hacking instances on the same problem and at the same turn number, yielding 868 matched pairs. Within each group, we count the fraction of (hack, non-hack) pairs where the hack turn projects higher on the Assistant Axis than the non-hack turn, and pool those counts across groups, resulting in an AUROC score.

The Assistant Axis vs. Random Controls

As I mentioned in the intro, it turns out that the Assistant Axis is correlated with reward-hacking positively, not negatively, with an AUROC of 0.71: in 71% of matched comparisons between a reward-hacking turn and a non-reward-hacking turn, the reward-hacking turn has a higher projection onto the Assistant Axis, as measured by activations in the final answer.

This is the opposite of what’s predicted by the initial story of “the model stops being the assistant when it cheats”. But how much is this really telling us about the Assistant Axis? Is there really something specific about “assistant-ness” that predicts a model’s propensity to reward-hack, or is there just not much meaningful going on at all?

As it turns out, AUROC scores for 95% of randomly-generated equal-norm directions range between 0.14 and 0.87, and 30% of these random directions predict reward-hacking at least as well or better than the Assistant Axis.

So why do random directions do so well? A random direction captures only a small fraction of the gap between hacking and non-hacking activations, but it also captures only a small fraction of the variation within each group. What matters is the gap relative to that variation. Here, the groups are strongly separated and the within-group variance is concentrated in very few directions. Together, those properties make the separation visible even along many random directions.

To test the role of that concentration, I added Gaussian noise equally in all 2,880 dimensions. The noise increases variance along the mean-difference direction by only 10%, but increases total variance by a factor of 39. AUROC along the mean-difference direction barely changes, from 0.94 to 0.91, while the middle 95% of random-direction AUROCs contracts from 0.14–0.87 to 0.41–0.59.

The groups remain easy to distinguish along their mean difference, but random directions now perform much closer to chance. By itself, strong separation doesn’t explain the original random-direction results; the concentration of variance matters too.

The final-answer token region result does also vary by benchmark, with the Assistant Axis scoring 0.80 on problems derived from ImpossibleBench, but 0.41 on EvilGenie-derived ones (albeit with a sample of only 28 hacks across 11 problems).

The reasoning-trace region does offer somewhat stronger evidence for the Assistant Axis: its AUROC is 0.696, and only 5.4% of random directions separate the classes at least as well (4.4% with 2,000 random directions, 6.7% after adjusting for token length). The prompt region shows essentially no separation, though.

So while hacking and non-hacking activations do differ along the Assistant Axis, it’s not necessarily very informative or unusual on its own.

The Schemer Axis

The Assistant Axis seemed like somewhat of a dead end as far as digging into reward-hacking. So I decided to try directly extracting an axis that is actually about scheming.

[results tbd]