1. [Publications](/publications)
2. DoRA: Weight-Decomposed Low-Rank Adaptation
 
 # DoRA: Weight-Decomposed Low-Rank Adaptation

  ![](/sites/default/files/styles/wide/public/publications/teaser_wide_v2.png?itok=SvQN940Z)

 In this [ICML'24 Oral](https://icml.cc/virtual/2024/events/oral) paper, we first introduce a novel weight decomposition analysis to investigate the inherent differences between FT and LoRA. Aiming to resemble the learning capacity of FT from the findings, we propose Weight-Decomposed LowRank Adaptation (DoRA). DoRA decomposes the pre-trained weight into two components, *magnitude* and *direction*, for fine-tuning, specifically employing LoRA for directional updates to efficiently minimize the number of trainable parameters. By employing DoRA, we enhance both the learning capacity and training stability of LoRA while avoiding any additional inference overhead. DoRA consistently outperforms LoRA on fine-tuning LLaMA, LLaVA, and VL-BART on various downstream tasks, such as commonsense reasoning, visual instruction tuning, and image/video-text understanding. Code available at [this https URL](https://github.com/NVlabs/DoRA).



 ## Authors



Shih-Yang Liu (NVIDIA, HKUST)

Chien-Yi Wang (NVIDIA)

[Hongxu Danny Yin](/person/danny-yin)

[Pavlo Molchanov](/person/pavlo-molchanov)

[Frank Wang](/person/frank-wang)

Kwang-Ting Cheng (HKUST)

[Min-Hung Chen](/person/min-hung-chen)

 

 

 ## Publication Date



Monday, July 22, 2024

 

 ## Published in



[International Conference on Machine Learning (ICML) 2024](https://icml.cc/Conferences/2024)

 

 ## Research Area



[Artificial Intelligence and Machine Learning ](/research-area/machine-learning-artificial-intelligence)

[Computer Vision](/research-area/computer-vision)

[Generative AI](/research-area/generative-ai)

[Natural Language Processing](/research-area/natural-language-processing)

 

 

 ## External Links



[Project Website](https://nbasyl.github.io/DoRA-project-page/)

[arXiv](https://arxiv.org/abs/2402.09353)

[Code](https://github.com/NVlabs/DoRA)

[NVIDIA Tech Blog](https://developer.nvidia.com/blog/introducing-dora-a-high-performing-alternative-to-lora-for-fine-tuning/)