Understanding and Enforcing Weight Disentanglement in Task Arithmetic
Abstract
Task arithmetic lacks theoretical explanation for its success, but the proposed OrthoReg method addresses this by promoting weight disentanglement through enforced orthogonality in weight updates during fine-tuning.
Task arithmetic provides an efficient, training-free way to edit pre-trained models, yet lacks a fundamental theoretical explanation for its success. The existing concept of ``weight disentanglement" describes the ideal outcome of non-interfering task composition but does not reveal its underlying cause. Crucially, what intrinsic properties of the pre-trained model (ฮธ_0) or the task vectors (ฯ_t) enable this disentanglement remains underexplored. In this paper, we introduce Task-Feature Specialization (TFS), a model's ability to allocate distinct internal features to different tasks, as the fundamental principle. We first prove that TFS is a sufficient condition for weight disentanglement. More importantly, we find that TFS also gives rise to an observable geometric consequence: weight vector orthogonality. This positions TFS as the common cause for both the desired functional outcome (disentanglement) and a measurable geometric property (orthogonality). This relationship provides the key insight for our method: since the abstract TFS property is intractable to enforce directly, we can instead promote weight disentanglement by shaping its concrete geometric consequence, orthogonality. Therefore, we propose OrthoReg, a simple and effective regularization method that actively enforces an internal orthogonal structure on weight updates (ฮW) that constitute ฯ_t during fine-tuning. And we theoretically prove that OrthoReg promotes disentanglement. Extensive experiments demonstrate that OrthoReg consistently and significantly enhances the performance of various task arithmetic methods. Code is available at https://github.com/RL-MIND/OrthoReg{https://github.com/RL-MIND/OrthoReg}.
Community
Hi everyone! ๐ค We are thrilled to share our latest work accepted at CVPR 2026: "Understanding and Enforcing Weight Disentanglement in Task Arithmetic".
If you are interested in Model Merging and Task Arithmetic, here is a quick overview of what we did and how you can use it!
๐ก The Core Problem: Why does Task Arithmetic work?
While task arithmetic (adding/subtracting task vectors) is an elegant, training-free way to edit models, its underlying mechanism has remained unclear. We theoretically prove that Task-Feature Specialization (TFS) is a sufficient condition for weight disentanglement. More importantly, TFS naturally leads to an observable geometric property: Weight Vector Orthogonality.
๐ ๏ธ Our Solution: OrthoReg
Since enforcing abstract feature specialization is intractable, we propose OrthoRegโa simple, plug-and-play regularization term added to the standard fine-tuning loss. It actively enforces column-wise orthogonality on the weight updates (\Delta W) during fine-tuning.
The loss is incredibly simple to implement:
๐ Key Results
- Consistent Boosts: Significantly improves the performance of various task arithmetic baselines (Non-linear FT, TTA, ATT-FT, LoRA) across ViT-B-32, ViT-B-16, and ViT-L-14.
- High Efficiency: Achieves the disentanglement benefits of Tangent Task Arithmetic (TTA) but without the massive computational overhead of Jacobian calculations.
- Cleaner Forgetting: Substantially improves task negation (making a model forget a specific task) with minimal side effects on general capabilities.
๐ Resources
We believe in open science and have released all our artifacts:
- ๐ป GitHub Repository: RL-MIND/OrthoReg.
- ๐ค Model Checkpoints: RL-MIND/OrthoReg_checkpoints (All fine-tuned checkpoints across 8 tasks and 6 modes are hosted right here on Hugging Face!).
We would love to hear your thoughts, feedback, or answer any questions you might have. Feel free to drop a comment below! ๐
Get this paper in your agent:
hf papers read 2604.17078 Don't have the latest CLI?
curl -LsSf https://hf.co/cli/install.sh | bash Models citing this paper 2
Datasets citing this paper 0
No dataset linking this paper
Spaces citing this paper 0
No Space linking this paper
Collections including this paper 0
No Collection including this paper