siddharthmb/2026.TA.gemma2_2b_gemmascope_transcoders_instruct_ft_all

GemmaScope transcoders re-fine-tuned to instruction-tuned hidden states, all 26 layers (gemma-2-2b)

What this is. The 26 GemmaScope transcoders of google/gemma-2-2b (width_16k, the average_l0_76-nearest folder per layer) re-fine-tuned so that each still imitates the base MLP, transcoder(x) ≈ MLP_base(x), but accurately when x is the instruction-tuned model's (google/gemma-2-2b-it) post-ln2 hidden state, with the sparsity (L0, active features per token) set to the level the pretrained transcoders have on base-model inputs. One file per layer, finetuned_layer_<L>.safetensors, holding the full transcoder state: W_enc, W_dec, b_enc, b_dec and the JumpReLU threshold. It extends the three-layer set siddharthmb/2026.TA.gemma2_2b_gemmascope_transcoders_instruct_ft_L0-24-25 to every layer.

The problem it fixes. Circuit tracing of the base-to-instruct gap feeds the base dictionary the instruct model's hidden states, a distribution shift that raises reconstruction error (FVU, fraction of variance unexplained; lower is better) at every layer, by 5 to 13 % through the middle of the stack and far more at the ends (layer 0 +33 %, layer 24 +32 %, layer 25 +104 % on chat; the same on web text), and moves L0 off the base operating point. In an attribution graph the lost fidelity is error-node mass.

How the weights were made, in two steps. (1) A reconstruction-only fine-tune of all 26 layers on about 2M instruct chat tokens (loss MSE(transcoder(x), MLP_base(x)), x from the instruct model, Adam at lr 1e-4, threshold frozen, four Slurm jobs by layer range). It cuts FVU at every layer but, with the threshold frozen, lets L0 drift above the base level on 23 layers. (2) A calibration that scales each layer's JumpReLU threshold vector by one scalar so that the layer's L0 on 200k held-out instruct tokens equals the base-input L0 of the pretrained transcoder on that layer (scales 0.845 to 1.292): the refit of the gate to the original activation frequency that the paper uses for its feature-overlap check, one number per layer, so the features' relative activations stay as trained. The per-layer L1 sparsity penalty that pinned L0 in the three-layer set does not transfer: 1e-3 holds at layers 24 and 25 and over-sparsifies layers 4 to 23 (L0 32 to 65 against base 54 to 98), and coefficients interpolated per layer still undershoot; the calibration lands on the base L0 exactly at a lower FVU cost (section 2.5 of the writeup).

The result. On 200k held-out instruct chat tokens, L0 equals the base level at every layer and FVU is 11 % to 31 % below the pretrained transcoders on the same inputs (mean 16 %), at or below their base-input FVU on layers 0 to 21; layers 22 to 25 stay above it (layer 25: 0.224 against 0.153). On 0.5M further chat tokens the picture is the same (12 % to 32 % FVU, L0 within 0.8 of the base level). On 0.5M held-out web tokens, which neither step saw, the gain is at the endpoints (-15 % at layer 0, -8 % at layer 24, -22 % at layer 25) and within 4 % elsewhere: the mid-layer gains are specific to the chat distribution. On base inputs the set is 2 % (chat) and 11 % (web) worse than the pretrained transcoders, so it is a dictionary for the instruct model's hidden states, not a replacement on the base model. The encoder rows barely rotate (mean cosine 0.998 to 0.9997 per layer, at most 0.6 % of a layer's features below 0.99), so the feature examples collected on the original weights stay valid.

How to use

The files patch onto the pretrained GemmaScope set at load time. With the transcoder-adapters repo, any GemmaScope-loading tool takes this repo id directly; the loader copies the four weight tensors and the threshold of every layer:

# Attribution graphs (overlay kind; the same two flags on --graph_kind fused)
uv run --extra viz python -m analysis.attribution.graphs.build_graphs --graph_kind overlay \
  --finetuned_transcoder_dir siddharthmb/2026.TA.gemma2_2b_gemmascope_transcoders_instruct_ft_all --finetuned_layers all ...

# Base feature examples on the instruct model's stream
uv run python -m analysis.features.collect --source gemmascope \
  --finetuned_transcoder_dir siddharthmb/2026.TA.gemma2_2b_gemmascope_transcoders_instruct_ft_all --finetuned_layers all ...

# FVU/L0 measurement with base and instruct inputs
uv run --extra viz python -m experiments.transcoder_input_shift.scripts.transcoder_input_shift --sources base instruct --layers all --n_tokens 500000 \
  --finetuned_transcoder_dir siddharthmb/2026.TA.gemma2_2b_gemmascope_transcoders_instruct_ft_all --finetuned_layers all

# A self-contained circuit-tracer transcoder set (config.yaml + layer_<L>.safetensors, 7.4 GB)
uv run --extra viz python -m analysis.attribution.export_finetuned_transcoder_set --finetune_dir siddharthmb/2026.TA.gemma2_2b_gemmascope_transcoders_instruct_ft_all --finetuned_layers all --output_dir <dir>

Or by hand: load the GemmaScope TranscoderSet, then for each layer copy finetuned_layer_<L>.safetensors's W_enc, W_dec, b_enc, b_dec into the transcoder and threshold into its JumpReLU.

How it was produced

Produced by python -m experiments.transcoder_input_shift.scripts.finetune_transcoder_shift from transcoder-adapters at commit 02e6d0b7 (the fine-tune jobs at 86362d58), on the Stanford NLP cluster's jag-standard partition (48 GB cards), 2026-10-14.

Exact commands (from the run manifests and Slurm logs; --gemmascope_width width_16k --gemmascope_l0 average_l0_76 are the defaults):

# Step 1: reconstruction-only fine-tune, four jobs by layer range (Slurm 17862860, 17862861, 17862862, 17862863; 12 to 17 min each)
uv run --extra viz python -m experiments.transcoder_input_shift.scripts.finetune_transcoder_shift --layers 0 1 2 3 4 5 6 --l1_coeff 0 --train_tokens 2000000 --n_tokens 200000 --eval_every_tokens 500000 --wandb
# ... and --layers 7 8 9 10 11 12 / 13 14 15 16 17 18 19 / 20 21 22 23 24 25; the 26 files gathered in one directory
# Step 2: the L0 calibration from those weights (Slurm 17865143; 9.5 min, no training)
uv run --extra viz python -m experiments.transcoder_input_shift.scripts.finetune_transcoder_shift --layers all --l1_coeff 0 --train_tokens 0 --n_tokens 200000 \
  --finetuned_transcoder_dir <step 1's 26 files> --finetuned_layers all --calibrate_l0 experiments/transcoder_input_shift/results/exp1_all_layers_results.json --wandb
# The base-input L0 targets per layer: Experiment 1 of the writeup (Slurm 16113808)
uv run --extra viz python -m experiments.transcoder_input_shift.scripts.transcoder_input_shift --sources base instruct --layers all --n_tokens 500000 --wandb

• Report: finetune_report.json (step 2: before is step 1's result, after the calibrated set, config.threshold_scale the scale per layer); finetune_report_reconstruction_only.json (step 1, the four jobs merged, with the eval every 0.5M tokens); gemmascope_config.json (the exact per-layer average_l0_* selection); encoder_drift.json; run_manifest.json (step 2).
• Build logs: logs/finetune_transcoder_shift/20261014_030201_1786286{0,1,2,3}.out (step 1), logs/finetune_transcoder_shift/20261014_064954_17865143.out (step 2) in the repo checkout.

Details

• Training data: the validation split of siddharthmb/2026.transcoder-adapters.lmsys-chat-1m-splits rendered with the gemma-2-2b-it chat template, 512 tokens per sequence, 16 sequences per batch; the first fifth of the shuffled batches (about 200k valid tokens) held out for the evals and the calibration, the next 2M tokens trained on. Web text for the held-out check: science-of-finetuning/fineweb-1m-sample.
• Objective: per layer MSE(transcoder(x), MLP_base(x)), x = the instruct model's blocks.L.ln2.hook_normalized, the target from a patched base forward; both models frozen in bf16, the transcoders trained in fp32 with Adam at lr 1e-4, no sparsity term. Each layer's term is backpropagated on its own and the model forwards stop after the last requested block, which is what fits 13 trained layers beside the two models on a 48 GB card (1.1 GiB per layer of fp32 weights, gradients and Adam state).
• Calibration: a feature is active where its pre-activation exceeds its threshold, so at scale s the active (token, feature) pairs are those with pre-activation / threshold above s; the scale that leaves exactly L0_base × tokens of them is read off the sorted ratios in one pass over the held-out tokens (Finetune.calibrate_thresholds). Threshold scales per layer: L0 1.049, L1 0.845, L2 0.937, L3 0.995, L4 1.051, L5 1.066, L6 1.057, L7 1.082, L8 1.292, L9 1.098, L10 1.173, L11 1.113, L12 1.169, L13 1.102, L14 1.156, L15 1.044, L16 1.139, L17 1.116, L18 1.216, L19 1.214, L20 1.242, L21 1.194, L22 1.244, L23 1.161, L24 1.243, L25 1.243.
• Conventions matched to the paper (arXiv 2602.20904, Transcoder Adapters for Model Diffing): training on the fine-tune's own distribution, Adam at the learning rate its MLP fine-tuning skyline found best, the decoder-norm-weighted L1 as the sparsity term where one was tried, and its refit of the gate to the original activation frequency (Appendix "Feature Activation Overlap") as the calibration.
• L0 selection: the average_l0_76-nearest folder per layer, the project's convention, which differs from the canonical mntss/gemma-scope-transcoders selection on 18 of 26 layers; the project's own feature examples (siddharthmb/2026.TA.features_gemma-2-2b_gemmascope_width_16k_average_l0_76_ms100000_ml1024_tk2_h12ad59325ffd) use the same selection and stay valid for this set (the encoders barely moved).

Results

Held-out instruct chat tokens (200k), per layer: the base-input numbers are the pretrained transcoder on base-model hidden states (the target), "before" the pretrained transcoder on instruct hidden states, "after" this set on instruct hidden states.

layer	base L0	L0 after	base-input FVU (target)	FVU before	FVU after	ΔFVU	threshold scale
0	80.2	80.2	0.082	0.108	0.075	-31%	1.049
1	63.1	63.1	0.310	0.312	0.255	-18%	0.845
2	89.8	89.8	0.279	0.296	0.235	-21%	0.937
3	93.9	93.9	0.270	0.296	0.246	-17%	0.995
4	84.3	84.3	0.296	0.318	0.261	-18%	1.051
5	85.5	85.5	0.365	0.389	0.321	-18%	1.066
6	97.5	97.5	0.353	0.391	0.316	-19%	1.057
7	70.0	70.0	0.379	0.404	0.331	-18%	1.082
8	90.6	90.6	0.358	0.402	0.326	-19%	1.292
9	70.7	70.7	0.422	0.434	0.373	-14%	1.098
10	84.9	84.9	0.456	0.453	0.402	-11%	1.173
11	60.6	60.6	0.486	0.487	0.431	-12%	1.113
12	58.8	58.8	0.500	0.535	0.473	-12%	1.169
13	89.9	89.9	0.452	0.476	0.419	-12%	1.102
14	81.4	81.4	0.454	0.475	0.421	-11%	1.156
15	77.6	77.6	0.402	0.446	0.379	-15%	1.044
16	86.8	86.8	0.386	0.423	0.371	-12%	1.139
17	60.4	60.4	0.406	0.440	0.387	-12%	1.116
18	53.6	53.6	0.340	0.373	0.327	-13%	1.216
19	82.4	82.4	0.309	0.344	0.302	-12%	1.214
20	80.4	80.4	0.324	0.357	0.316	-11%	1.242
21	55.7	55.7	0.347	0.386	0.342	-11%	1.194
22	60.4	60.4	0.343	0.387	0.344	-11%	1.244
23	62.9	62.9	0.310	0.365	0.317	-13%	1.161
24	56.4	56.4	0.210	0.281	0.228	-19%	1.243
25	64.8	64.8	0.153	0.317	0.224	-29%	1.243

The same set measured on 0.5M chat tokens and 0.5M held-out web tokens (FVU; L0 of the pretrained transcoder on base inputs against this set on instruct inputs):

layer	chat: original on base inputs	original on instruct inputs	calibrated on instruct inputs	L0 original base / calibrated instruct	web: original on base inputs	original on instruct inputs	calibrated on instruct inputs	L0 original base / calibrated instruct
0	0.082	0.110	0.075	80.2 / 81.0	0.075	0.098	0.083	75.2 / 73.7
1	0.310	0.311	0.253	63.1 / 63.6	0.327	0.336	0.332	62.3 / 64.5
2	0.279	0.298	0.234	89.8 / 90.6	0.275	0.293	0.288	80.9 / 81.2
3	0.270	0.297	0.245	93.9 / 94.0	0.285	0.308	0.307	89.6 / 87.5
4	0.296	0.320	0.260	84.3 / 84.2	0.289	0.313	0.311	81.9 / 81.5
5	0.365	0.389	0.316	85.5 / 85.8	0.357	0.385	0.384	90.2 / 93.3
6	0.353	0.392	0.312	97.5 / 97.8	0.343	0.388	0.379	99.7 / 101.8
7	0.379	0.404	0.325	70.0 / 70.4	0.381	0.422	0.418	72.9 / 76.6
8	0.358	0.403	0.322	90.6 / 90.7	0.348	0.407	0.396	93.1 / 92.9
9	0.422	0.434	0.366	70.7 / 70.5	0.398	0.450	0.455	75.0 / 74.9
10	0.456	0.452	0.394	84.9 / 84.8	0.435	0.472	0.484	89.6 / 87.3
11	0.486	0.483	0.420	60.6 / 60.6	0.465	0.512	0.512	59.6 / 63.2
12	0.500	0.537	0.467	58.8 / 58.8	0.486	0.549	0.543	60.6 / 58.1
13	0.452	0.476	0.412	89.9 / 89.6	0.428	0.498	0.485	94.7 / 87.1
14	0.454	0.475	0.413	81.4 / 80.9	0.417	0.483	0.476	87.6 / 76.1
15	0.402	0.447	0.373	77.6 / 77.1	0.362	0.434	0.416	84.5 / 72.1
16	0.386	0.424	0.366	86.8 / 86.1	0.357	0.411	0.409	97.2 / 80.7
17	0.406	0.441	0.383	60.4 / 59.9	0.383	0.424	0.422	65.2 / 55.8
18	0.340	0.374	0.323	53.6 / 53.3	0.319	0.348	0.349	61.1 / 50.2
19	0.309	0.346	0.300	82.4 / 82.0	0.296	0.329	0.337	96.8 / 76.8
20	0.324	0.358	0.313	80.4 / 80.1	0.316	0.346	0.355	95.8 / 74.6
21	0.347	0.388	0.339	55.7 / 55.5	0.339	0.374	0.379	66.6 / 52.0
22	0.343	0.386	0.340	60.4 / 60.3	0.354	0.399	0.403	74.5 / 57.4
23	0.310	0.363	0.313	62.9 / 63.1	0.345	0.409	0.400	74.1 / 58.9
24	0.210	0.277	0.223	56.4 / 56.7	0.270	0.344	0.315	61.5 / 50.2
25	0.153	0.312	0.219	64.8 / 64.7	0.179	0.408	0.319	71.4 / 66.3

Links

• Writeup: experiments/transcoder_input_shift/writeup.md, section 2.5 (the measurement of the shift is section 1, the three-layer fine-tune sections 2.1 to 2.4).
• Weights & Biases (project siddharth-stanford/transcoder-feature-collection): step 1 ezyihirs, us8k1cnc, 7osd0tnm, ypcyui8d; step 2 1rz594b6; the measurements k2e1g3kx (chat), 8cwr5o0s (web), 0zdyov2w (the pretrained set on web).
• Three-layer set: siddharthmb/2026.TA.gemma2_2b_gemmascope_transcoders_instruct_ft_L0-24-25 (layers 0, 24, 25 with the L1 penalty recipe, July 2026).
• Paper: Transcoder Adapters for Model Diffing.
• Code: https://github.com/Sid-MB/transcoder-adapters

Cluster paths

Under $LARGE_ARTIFACTS_DIR/transcoder-adapters/ (/nlp/scr/siddharth on the NLP cluster):

• step 2, the files here: transcoder_input_shift_finetune/ft_gemmascope_width_16k_average_l0_76_L0-1-2-3-4-5-6-7-8-9-10-11-12-13-14-15-16-17-18-19-20-21-22-23-24-25_20261014_070646_17865143/
• step 1: transcoder_input_shift_finetune/ft_gemmascope_width_16k_average_l0_76_L0-1-2-3-4-5-6_20261014_043225_17862860/, ..._L7-8-9-10-11-12_20261014_044444_17862861/, ..._L13-14-15-16-17-18-19_20261014_045745_17862862/, ..._L20-21-22-23-24-25_20261014_051344_17862863/; gathered in transcoder_input_shift_finetune/all_layers_nopenalty_20261014_17862860-63/
• the measurements: transcoder_input_shift/gemmascope_width_16k_average_l0_76_base-instruct_20261014_073346_17865707/ (chat), ..._20261014_073555_17865708/ (web), ..._20261014_053044_17862865/ (the pretrained set on web)
• the materialised circuit-tracer set: finetuned_transcoder_sets/gemma2_2b_width16k_l0_76nearest_ftLall_calibrated/
• this upload's staging copy: hf_upload/2026.TA.gemma2_2b_gemmascope_transcoders_instruct_ft_all/