JANUS-LoRA

JANUS-LoRA

ICML 2026

A Balanced Low-Rank Adaptation for Continual Learning

Cheng Chen1 Pengpeng Zeng2 Yuyu Guo Lianli Gao1 Hengtao Shen2 Jingkuan Song2,3,*

* Corresponding author

1 UESTC logo

School of Computer Science and Engineering, UESTC, Chengdu, China

2

School of Computer Science and Technology, Tongji University, Shanghai, China

3 Shanghai Innovation Institute logo

Shanghai Innovation Institute, Shanghai, China

Code PDF
Overview of the JANUS-LoRA framework
Online Estimation estimates the protected subspace, Gradient Rectification makes the LoRA update follow the safe direction, and Decoupled Margin Loss restores feature-level plasticity.
01

From Orthogonality to Diagnosis

Sufficient condition

Old outputs remain unchanged when

ΔW · Xpast = 0

Orthogonality is a sufficient condition for zero interference: updates in the null space do not change historical activations.

LoRA complication

ΔW = BΔA + ΔBA

LoRA does not optimize the full update directly. Independent updates to A and B can make the composite update deviate from the safe direction.

Two observed failures

The paper diagnoses parameter-level misalignment and feature-space encroachment as the two coupled failures behind forgetting.

Null-space violation scores with and without Gradient Rectification
GR reduces the measured null-space violation across K/V layers.
Feature-space scatter without DML
Feature-space scatter with DML
DML lowers feature invasion danger from 0.405 to 0.202.
02

Method: Resolve Both Failures

Online Estimation

Maintains an orthonormal basis V for the protected historical subspace without storing old samples.

Gradient Rectification

Converts the ideal safe update ΔWsafe into corrected LoRA factor updates ΔA and ΔB.

Decoupled Margin Loss

Separates new features from old prototypes so plasticity is preserved under the orthogonality constraint.

JANUS-LoRA method overview
03

Experiments as an Evidence Chain

Overall effectiveness

Compare JANUS-LoRA with CL and LoRA-based baselines under increasing ImageNet-R task counts.

Conclusion: the method keeps the highest T=20 MAA at 77.11%.

Mechanism isolation

Use ablation and plug-in tests to separate the roles of OE, GR, and DML.

Conclusion: GR fixes parameter interference, while DML recovers plasticity.

Robustness and cost

Check cross-dataset transfer, online estimation behavior, and cumulative training time.

Conclusion: the balance principle generalizes without excessive runtime cost.

Validation of Long-Sequence Effectiveness

ImageNet-R main result table
Goal Test whether JANUS-LoRA remains effective as ImageNet-R is split into longer task sequences. Conclusion JANUS-LoRA achieves the best T=20 MAA, reaching 77.11%.

Validation of Component Contributions

Ablation table
Goal Verify that OE, GR, and DML each contribute to the final stability-plasticity balance. Conclusion The full model gives the best ACC/MAA, while removing modules weakens either stability or plasticity.

Validation of GR General Applicability

GR improvement comparison
Goal Check whether GR directly addresses LoRA's composite-update misalignment. Conclusion As a plug-in correction, GR improves LoRA-GPM by +2.25 ACC and +6.93 BWT.

Validation of Sustained Continual Performance

ImageNet-R trajectory
Goal Inspect whether the advantage persists over the continual learning trajectory. Conclusion JANUS-LoRA keeps a stronger accuracy trajectory instead of only improving the final task.

Validation of Cross-Dataset Generalization

Generalization table
Goal Test whether the same design transfers beyond ImageNet-R. Conclusion JANUS-LoRA leads on CIFAR-100, ImageNet-100, and DomainNet with 88.68/92.58, 92.47/94.32, and 73.82/79.67 ACC/MAA.

Validation of OE in Single-Pass Training

OE and GPM comparison
Goal Determine whether online subspace estimation can replace heavier GPM-style estimation. Conclusion OE yields a +3.45 MAA gain under single-epoch training while avoiding stored historical data.

Validation of Training Efficiency

Cumulative training time
Goal Measure whether the stronger constraints introduce impractical training overhead. Conclusion JANUS-LoRA stays efficient, with cumulative training time reported as 1197.8s.
04

Citation

@inproceedings{chen2026januslora,
  title     = {JANUS-LoRA: A Balanced Low-Rank Adaptation for Continual Learning},
  author    = {Chen, Cheng and Zeng, Pengpeng and Guo, Yuyu and Gao, Lianli and Shen, Hengtao and Song, Jingkuan},
  booktitle = {International Conference on Machine Learning},
  year      = {2026}
}