RESEARCH NOTE · SEPTEMBER 2026

When Similar Representations Don't Lead to Similar Behaviour: A Cross-Lingual Steering Puzzle

What a mismatch between steering-vector geometry and behaviour changed about how I think about multilingual alignment.

The puzzle

When I started my MSc dissertation, I had a fairly simple question: if an activation steering intervention works in English, will it also work in other languages?

I used Contrastive Activation Addition (CAA) to construct steering directions from English examples, then applied the same interventions to Italian, Chinese, Korean, and Kazakh. I expected that if different languages represented the same behavioural concept in similar internal directions, the intervention should transfer reasonably well.

At first, the geometry seemed to support that intuition.

Across several languages, steering vectors extracted independently from different languages were highly similar to the English direction. In some settings, cosine similarity was above 0.9. That looked like evidence for a shared representation.

But the behavioural results were much less uniform.

Some models transferred the intervention reliably across languages. Others did not. And the case that changed how I think about this came from comparing two models on the same task. For Italian and Chinese, English-derived truthfulness directions were highly similar to the language-specific directions in both Qwen and OLMo3, above 0.94 in each case. Yet only OLMo3 showed reliable truthfulness transfer.

If internal representations look aligned across languages, why does behaviour still diverge?

What I tested

The main experiments focused on sycophancy: whether a model changes its answer to agree with a user's stated opinion.

I constructed English steering vectors and applied them across five languages: English, Italian, Chinese, Korean, and Kazakh. I tested three open-weight model families, Qwen3.5-9B, Gemma4-12B-IT, and OLMo3-7B-Instruct, and compared behavioural changes after steering. Experiments ran on the Stanage HPC cluster, with bootstrap confidence intervals for all effects and Gaussian and permutation control vectors for the sycophancy experiments.

To understand whether cross-lingual behavioural transfer was related to representation geometry, I also extracted language-specific steering vectors and measured their cosine similarity to the English direction.

I then extended the analysis beyond sycophancy using TruthfulQA, constructing separate English-derived truthfulness vectors and asking whether transfer patterns that held for one behaviour would hold for another.

The result was not a clean yes or no.

The result that changed how I think about steering

For Qwen and Gemma, English-derived sycophancy steering transferred reliably to all four target languages. Cosine similarities ranged from 0.918 to 0.987, yet geometry did not track effect magnitude. In Qwen, Korean was more similar to English than Kazakh was (0.950 vs 0.916 at the sycophancy layer), but the two produced almost identical effects (19.1 vs 19.3 pp). Geometry was not useless, but it was not a reliable guide to how much an intervention would do.

OLMo3 made the limits of geometry harder to ignore, though not in the way I first expected. For sycophancy, its cross-lingual similarities were lower than Qwen's and Gemma's (0.382 to 0.838), and English-derived steering produced no reliable effect outside English. The clearer case came from TruthfulQA. For Italian and Chinese, truthfulness-vector similarities exceeded 0.94 in both Qwen and OLMo3 (Qwen L18: IT 0.984, ZH 0.980; OLMo3 L14: IT 0.948, ZH 0.960). Qwen showed no reliable transfer in either language, while OLMo3 improved truthfulness by about 9 pp in both, with confidence intervals excluding zero. Comparable geometry in the same languages and on the same task produced divergent outcomes.

This sharpened the central finding: high cosine similarity is not sufficient for behavioural transfer. Whether it is even necessary is less clear. But the relationship between representational similarity and behavioural effect was not consistent enough across models, languages, and tasks to treat geometry as a reliable proxy for transferability.

What I initially got wrong

My initial intuition was that cosine similarity could act as a proxy for transferability.

If two language-specific steering vectors point in almost the same direction, then perhaps they encode the same concept in the same way, and therefore the same intervention should work similarly in both languages.

The experiments made that interpretation harder to defend.

Cosine similarity tells us that two directions are geometrically aligned. It does not tell us whether they are equally causally important for downstream behaviour.

A model might represent a behavioural feature similarly across languages but use that feature differently later in the network. Or the relevant information might be distributed across a broader subspace that a single steering vector does not capture.

This suggests that several questions need to be separated:

• Can we recover a similar direction across languages?
• Is that direction geometrically stable?
• Does intervening on it produce the same behavioural effect?

Those are related questions, but they are not equivalent.

Four explanations I now want to test

Metric limitation. Cosine similarity between two vectors may hide meaningful differences in the larger representational subspace. The metric captures directional alignment between summary vectors but says nothing about the structure of the subspace those vectors sit in.

Subspace mismatch. Two languages may encode the same concept partly in shared directions but rely on different combinations of features when producing behaviour. A single steering vector extracted from English may not span the dimensions that matter for Chinese or Korean.

Downstream divergence. The relevant representation may be present in both languages at the layer where we intervene, but later layers may transform or route that information differently before it affects the final output. The intervention lands in the same place but does not propagate the same way.

Training exposure. OLMo3's dominant pretraining sources were filtered for English, while its instruction-tuning stages introduced multilingual data. The divergence may reflect what each model was exposed to, and at which stage, rather than anything visible in the geometry at the intervention layer. Because OLMo3 releases its training data and checkpoints stage by stage, this explanation is testable.

These explanations imply different experiments. Metric limitation points toward subspace-level analysis rather than pairwise cosine. Subspace mismatch calls for probe transfer and behaviour-relevant subspace comparisons across languages. Downstream divergence points toward causal activation patching and layer-wise localisation to find where the divergence actually begins. Training exposure points toward measuring language-specific exposure in the training data and comparing checkpoints across training stages.

Where I want to go next

The next step is not simply to collect more cross-lingual steering results.

I want to understand where the divergence becomes behaviourally important. That means moving from describing transfer failures to explaining them: testing probe transfer across languages, comparing behaviour-relevant subspaces, localising where interventions stop generalising, and tracing whether the models' training exposure accounts for what the geometry cannot.

I am also interested in whether the same problem appears outside activation steering. If alignment methods are evaluated in one language or one task, how much confidence should we have that they will remain reliable under a distribution shift?

For me, that is now the broader research question.

The most interesting lesson from this project was not that multilingual steering sometimes fails. It was that an intervention can look internally consistent while behaving inconsistently, and that two models can share comparably high cross-lingual geometry and still transfer differently.

That gap is what I want to explain next.