PEFT-Arena: Understanding Parameter-Efficient Finetuning from a Stability-Plasticity Perspective

PEFT evaluation should ask not only what improves, but also what is forgotten and why.

Parameter-efficient finetuning is usually sold as an affordable way to adapt large language models. PEFT-Arena asks a less comfortable question: when a PEFT method improves the target task, what pretrained capabilities does it trade away? We benchmark this target-retention trade-off, then diagnose it through weight-space geometry, activation-space geometry, and interpolation paths.

• Target-only evaluation is not comprehensive. Full finetuning can deliver strong target gains while erasing general capabilities.
• PEFT methods form different stability-plasticity frontiers. OFT often gives a better retention-adaptation balance than additive low-rank methods at comparable budgets.
• Forgetting is tied to geometry distortion. Weight-space profiles show where updates act; activation-space metrics show whether general representations remain structurally stable.
• Interpolation diagnoses overshoot. SFT checkpoints often move beyond the best target-retention point, and interpolation should follow the geometry of each PEFT parameterization.

Why target accuracy is not enough

PEFT is usually evaluated by a downstream score: did math accuracy improve, did medical QA improve, did reward optimization help? But this hides the other half of adaptation. A model can become better on the target distribution while becoming worse at following instructions, recalling factual knowledge, or solving general reasoning tasks.

This is the stability-plasticity dilemma. Plasticity asks how much the model learns from the new domain. Stability asks how much pretrained capability survives the update. PEFT-Arena treats both as first-class evaluation axes.

A PEFT method is not strong if it only learns the target task by spending pretrained capabilities.

The benchmark: read the table horizontally

PEFT-Arena evaluates Qwen2.5-7B and Llama3.2-3B-Instruct under supervised finetuning and RLVR with GRPO. We adapt to mathematical and medical reasoning, then evaluate retained general ability on IFEval, NQ, and BBH. Each cell below reports Target / General average accuracy.

The table should be read horizontally. Full FT often maximizes the target column, but it spends a large amount of general capability to get there. PiSSA is a more extreme failure mode: it can improve or even destabilize target performance while severely damaging general retention. OFT does not always maximize target accuracy, but it often preserves substantially more general ability for comparable target gains. RLVR behaves differently from SFT: at the evaluated horizon, it obtains target gains with much weaker general erosion.

From scores to mechanisms

The benchmark tells us what happens, but not why. Two PEFT methods can have similar target gains and very different forgetting. To explain this, PEFT-Arena looks inside the model from two complementary views: weight-space geometry and activation-space geometry.

Weight-space geometry: where does the update act?

In weight space, we decompose each pretrained weight matrix into its singular directions. This gives us a coordinate system for asking where the finetuned update acts.

We use two profiles. The retention profile measures how much the finetuned weight deviates along the pretrained singular structure. The adaptation profile measures where update energy is injected across pretrained directions. These profiles are descriptive: they show the inductive bias of a PEFT parameterization rather than serving as a complete theory of forgetting.

Full FT and PiSSA visibly disrupt the retention profile. LoRA, MiSS, PiSSA, and AdaLoRA often allocate update energy in spiky directions. PiSSA is especially unstable: it updates principal components strongly, but it also remains spiky across the rest of the spectrum. OFT is more structured because it updates through an orthogonal parameterization, which tends to preserve pretrained spectral geometry better.

Weight-space geometry tells us where a PEFT update acts, but not whether those directions matter for a capability distribution.

Capability-conditioned drift: does the data use those directions?

A large weight update is not automatically harmful. It matters whether general examples actually activate the changed directions. Capability-conditioned drift measures the effect of the update on pretrained activations from a dataset. We compute it separately for general data and target-domain data.

General CSD is associated with forgetting, while target CSD is not a clean monotonic predictor of target gain. This suggests that retention is more directly tied to perturbing general-distribution activations, whereas target adaptation depends on whether the change is task-aligned.

Activation-space geometry: did the representation structure survive?

Weight-space diagnostics still miss an important distinction: movement is not the same as damage. A representation can move through a nearly shared rotation while preserving distances, angles, and pairwise relationships. This is why raw activation drift can be misleading, especially for OFT.⁴

We therefore compare base-model and finetuned-model activations on the same general examples. The main question is whether the relational geometry among examples survives finetuning.

Procrustes residual and Gram distortion correlate with forgetting; CKA moves in the opposite direction and tracks retained general score. OFT can still move representations, but much of this movement is more rotation-like and less destructive. PiSSA shows strong non-isometric distortion, matching its severe forgetting.

The retention-side signal is not “how far activations move,” but whether their geometry is distorted beyond a benign rotation.

Interpolation: where along the path is stability lost?

A final checkpoint is only one point along the adaptation path. Interpolation asks whether the model moved farther than needed for its target gain. In SFT, intermediate points often recover general ability while preserving much of the target improvement. This reveals an overshoot region: the final checkpoint is not always the best target-retention operating point.

Across SFT runs, the rewinding path for Full FT and PEFT methods shows a common pattern: general ability recovers first, while target performance drops only later. This creates practical operating points that retain most of the target gain with substantially less general-capability damage. It also raises a pathwise question: is rewinding simply reversing the training trajectory, or does it trace a different route through the target-general plane? Comparing training checkpoints with interpolation points shows that the two paths are not the same. We call this an overshoot phenomenon because the final SFT checkpoint often moves past a better target-retention trade-off point, and interpolation reveals that the extra movement mainly hurts stability before it meaningfully helps plasticity.

The interpolation path must respect the PEFT parameterization. For additive methods such as Full FT, LoRA, and PiSSA, the natural path scales the effective update ΔW. For OFT, the update is represented by a skew-symmetric generator and a Cayley transform, so interpolation should scale the generator and stay on the orthogonal path. The difference is not only geometric: in the table below, the parameterization-aware path recovers much more general ability at a similar interpolation strength than dense-weight interpolation.

Layer-wise OFT rewinding is a post-hoc case study. OFT update strengths are not balanced across depth; later layers can receive much stronger rotations than early layers. SafeScale and MinScale rescale layer generators to reduce this imbalance and can recover general ability while preserving much of the target gain.

What we learn

• PEFT methods should be compared by target-retention trade-offs, not target score alone.
• OFT often gives a strong frontier because its parameterization better preserves geometry.
• PiSSA is a useful warning case: principal-component updates can severely disrupt general capabilities.
• Activation geometry clarifies why raw movement is not equivalent to forgetting.
• Interpolation turns final-checkpoint evaluation into a pathwise diagnosis.

Code and resources

The release includes training and evaluation entrypoints, benchmark result tables, weight-space spectral analysis, CSD diagnostics, activation-geometry scripts, and interpolation / rewinding utilities.

Repository contents.

• Training and evaluation entrypoints for SFT and RLVR.
• Benchmark result tables for target domains and general retention.
• Weight-space geometry analysis scripts.
• Activation-geometry and CSD diagnostics.
• Interpolation and post-hoc rewinding utilities.

bash setup_env.sh

python run.py train sft ...
python run.py train rl ...
python run.py eval ...
python run.py merge ...

Authors

Yangyi Huang^1,†, Ruotian Peng^2,†, Zeju Qiu³, Jiale Kang¹, Yandong Wen², Bernhard Schölkopf³, and Weiyang Liu^1,3,*.

¹The Chinese University of Hong Kong   ²Westlake University   ³Max Planck Institute for Intelligent Systems, Tübingen.

^†Equal contribution.   ^*Corresponding author.

Citation

@article{huang2026peftarena,
      title={PEFT-Arena: Understanding Parameter-Efficient Finetuning from a Stability-Plasticity Perspective},
      author={Huang, Yangyi and Peng, Ruotian and Qiu, Zeju and Kang, Jiale and Wen, Yandong and Sch\"olkopf, Bernhard and Liu, Weiyang},
      journal={arXiv preprint arXiv:2605.28819},
      year={2026}}