NoRA: Normalized Low-Rank Adaptation

LoRA's earliest optimization is decided entirely by the down-projection. That is where the scale problem lives, and where the fix belongs.

LoRA is the default way to adapt a large model cheaply, but how to regularize its training dynamics is still largely unexplored. Because LoRA initializes the up-projection to zero, the down-projection A alone governs the early trajectory. NoRA normalizes A along the rank dimension — every column unit-norm — which keeps the update exactly linear in the input and exactly mergeable, while removing the arbitrary per-coordinate scale imbalance that random initialization introduces. The same normalization applied only once at initialization already recovers most of the gain.

• The down-projection is an underused design dimension. With B⁽⁰⁾ = 0, A receives no gradient at step 0, so A⁽⁰⁾ fixes both the learning rates and the input subspace during the decisive early phase.
• Normalize along the rank dimension, not the input dimension. Column-wise normalization (Norm_r) lifts the SFT average from 29.3 to 37.2; row-wise normalization (Norm_k) does essentially nothing.
• Mergeability is preserved. Norm(A) depends only on the adapter parameters, not on the input, so Δy = αB Norm(A)x stays linear in x and folds back into W after training.
• Initialization does most of the work. NoRA-init normalizes once and then trains as plain LoRA, reaching 42.38 SFT average against LoRA's 37.93; the persistent constraint adds the rest (43.37).
• It is a preconditioner correction. LoRA is full finetuning with the gradient right-multiplied by P = α²A^⊤A. NoRA sets diag(P) = I and keeps it there.

The down-projection decides LoRA's early training

LoRA parameterizes the weight update of a pretrained matrix as ΔW = αBA, with A ∈ ℝ^r×k randomly initialized and B ∈ ℝ^d×r initialized to zero. The zero initialization is what preserves the pretrained model at step 0 — and it is also what makes the down-projection decisive. Writing G = ∂L/∂W for the full-weight gradient:

Existing work already shows that initialization matters a great deal here. PiSSA and MiLoRA build the initial subspace from the spectral structure of the pretrained weights and converge faster than random initialization — but they require an SVD, they modify the pretrained decomposition, and they can be fragile in reinforcement learning. Other lines of work change the scaling, the parameterization, or freeze the projections. None of them asks the narrower question: what happens if we simply regularize the magnitude of the down-projection?

It helps to put existing methods in one frame. Every low-rank method builds a latent feature φ(x) ∈ ℝ^r that is handed to the up-projection, Δy = αBφ(x):

From normalized latent features to a normalized projection

Our starting observation comes from Multi-head Latent Attention. MLA constructs its low-dimensional latent as φ(x) = Norm(Ax), and that normalized bottleneck noticeably stabilizes training and speeds up convergence. The obvious move — drop the same operation into LoRA — does not work. Normalizing Ax makes the transformation depend on the input, so the update becomes nonlinear in x and can no longer be folded back into the pretrained weight matrix. LoRA's mergeability is exactly the property nobody wants to give up.

So we move the normalization off the feature and onto the matrix. Each column a_j of A ∈ ℝ^r×k describes how input coordinate j is projected into the r-dimensional latent space. Modern transformers are pre-normalized, so the input x to each linear layer is already scale-controlled; what remains uncontrolled is the spread of those column norms. NoRA fixes them:

The transition is from MLA's input-dependent latent normalization, B Norm(Ax), to NoRA's normalized low-rank projection, B Norm(A)x.

NoRA-init and the block-identity variant

The normalization principle admits two ways of being applied. NoRA enforces it as a training constraint, re-normalizing A along the rank dimension throughout training. NoRA-init applies it once — A⁽⁰⁾ = Norm(A_init), B⁽⁰⁾ = 0 — and then trains with the standard LoRA parameterization, no re-normalization, no change to the optimizer. NoRA-init retains most of the benefit, which is itself the evidence that the early phase is where the scale problem does its damage.

There is also a deterministic way to satisfy the constraint. BIMI (Block Identity Matrix Initialization) builds A from repeated identity blocks: for k = br + q with 0 ≤ q < r,

This also explains an existing method. MiSS shares matrix shards, and its update Δy = αB[I_r, …, I_r]x is precisely LoRA with a fixed block-identity down-projection — whose columns are already unit-norm. MiSS is therefore a special case of NoRA in which A is frozen at A_fix. That predicts MiSS should show the same early-gradient signature as NoRA, and in the figure above it does. BIMI is what you get by keeping the construction but letting the projection train.

Why it works: LoRA has a hidden preconditioner

Take a gradient step of size η on B at initialization: ΔB = −ηαGA^⊤, while A receives no gradient and stays fixed. The induced change of the merged weight is

Reading P column by column makes the failure mode concrete. The column norms of A are per-coordinate learning rates:

A preconditioner should remove distortion, not add its own. Two consequences follow, and both are testable. Row normalization (Norm_k) leaves diag(P) random of order r/k, so it should bring no gain — and it does not. BIMI uses unit basis vectors as columns, giving the same diagonal through a completely different crosstalk pattern, so if the diagonal is what matters it should perform like normalized random initialization — and it does. That points at diag(P) as the decisive quantity.

Finally, why keep the constraint at all if initialization does most of the work? Because it adds two things. The loss becomes invariant to each column's scale, so gradients are tangential and, under gradient descent, norms can only grow while the effective step anneals automatically — the familiar behaviour of weight-normalized training. And diag(P) = I is enforced for the whole run on a compact set of directions, so the latent scale can neither collapse nor explode, while the update stays linear in x and merges exactly.

Which dimension should be normalized?

Given A ∈ ℝ^r×k, there are two normalizations to try: Norm_k normalizes each row, Norm_r normalizes each column A_:,j ∈ ℝ^r. The preconditioning argument says only the second one should matter, and only the second one does. Norm_r adds roughly seven points of average accuracy under every initialization distribution we tried; Norm_k adds nothing. The benefit is essentially independent of the initialization distribution, and BIMI — deterministic, no randomness at all — lands in the same place.

Pretraining: normalization prevents optimization collapse

We start from controlled MLA experiments, where normalizing the latent Ax consistently improves performance and accelerates convergence, and then carry the same principle over to the linear low-rank form. Under MLA, NoRA-init converges faster and scores better than the standard low-rank parameterization; it remains slightly behind directly normalizing Ax, but it keeps the BAx structure and with it the mergeability and deployment efficiency.

The MHA setting is where the difference becomes dramatic. Parameterizing every attention projection (q, k, v, o) with rank r = 64 low-rank factors, the standard parameterization collapses outright on LAMBADA and degrades badly elsewhere; its gradient norms fall to abnormally low levels during training. NoRA-init keeps gradient magnitudes well-scaled throughout and avoids the collapse entirely. Capacity alone does not determine optimization behaviour — under the same rank constraint, changing only the initialization of the projection changes the training dynamics.

Supervised finetuning: adaptation without spending retention

On Llama-3.2-3B across mathematical reasoning and code generation, NoRA gives the best overall SFT average among the compared methods, lifting LoRA's 37.93 to 43.37 — a gain of 5.44 points — with particularly clear improvements on GSM8K (50.94 → 61.63) and HumanEval (37.20 → 42.10). It also outperforms PiSSA, OFT, rsLoRA and MiSS on average, at the same 48.6M trainable parameters as LoRA.

The split between the two variants is informative. NoRA-init alone reaches 42.38, capturing most of the gap; maintaining the constraint through training adds the remaining point. Controlling the input-to-latent magnitude at initialization accounts for the bulk of the benefit, and persistent normalization contributes a further, smaller improvement.

Two further observations. NoRA outperforms MiSS (42.70) while keeping the down-projection trainable, so the block-identity construction is not doing anything a normalized trainable projection cannot. And the principle transfers: applying Norm_r to DoRA lifts its SFT average from 38.30 to 41.40 for NoRA-init and 43.34 for NoRA, which suggests rank-dimension normalization is a general property of low-rank adaptation rather than a LoRA-specific trick.

Retention comes along for free. NoRA's average change across MMLU, AGIEval and ARC-C is +0.02, against −0.56 for LoRA and −0.70 for MiSS — it improves adaptation without spending the pretrained model's general knowledge to do it.

RLVR: stable where spectral initialization is not

Under reinforcement learning with verifiable rewards on DeepSeek-R1-Distill-Qwen-1.5B, LoRA lifts the overall average from 41.0 to 42.8 and NoRA takes it to 44.4 — 3.4 points over the base model and 1.6 over LoRA — with the clearest gains on AMC, MATH500 and Minerva.

The more striking result is what happens to the spectral-initialization methods. PiSSA and MiLoRA are competitive under supervised finetuning, but here they degrade catastrophically: MiLoRA drops to 18.0 and PiSSA to 0.2, consistent with earlier reports of optimization instability for spectral initialization in RL. NoRA never touches the spectral structure of the pretrained weights, needs no SVD, and stays stable across both regimes.

What we learn

• LoRA's effectiveness depends not only on its rank but on the geometry and scale of its down-projection.
• The useful part of MLA's normalized latent can be transferred to LoRA without giving up linearity in x or exact mergeability — move the normalization from the feature to the matrix.
• Normalize along the rank dimension. The input dimension is the wrong axis, and the preconditioning view says why: only diag(P) is corrected by column normalization.
• Most of the benefit is set at initialization, because A's gradient vanishes with B. The persistent constraint buys stability, not the bulk of the gain.
• Methods that depend on the pretrained spectrum can be strong under SFT and fragile under RL. Normalization is cheap, distribution-agnostic, and holds up in both.

Settings

For NoRA we recommend α = r as the default, which puts the early gradient norm close to full finetuning. A larger scaling factor such as α = 2r can fit the loss better in some cases, but we observe increased forgetting of pretrained knowledge; given that trade-off, α : r = 1 : 1 is the configuration we use and recommend.

Authors

Jiale Kang^1,3, Ziyin Yue, Zheng Zhan², Yangyi Huang³, and Weiyang Liu^3,4,*.

¹Yuanshi Intelligence   ²Microsoft Research   ³The Chinese University of Hong Kong   ⁴Shenzhen Loop Area Institute.

^*Corresponding author.

Citation

@article{kang2026nora,
      title={Normalized Low-Rank Adaptation},
      author={Kang, Jiale and Yue, Ziyin and Zhan, Zheng and Huang, Yangyi and Liu, Weiyang},
      journal={arXiv preprint arXiv:XXXX.XXXXX},
      year={2026}}