Understanding LoRA: Low-rank Adaption of Large Language Models

LoRA (Low-Rank Adaptation) is one of the most powerful techniques when it comes to Fine-Tuning Large Language Models(LLMs).

Today I’ll clearly explain:

• What is Lora❓

• How does it works ❓

• Followed by a hands-on coding tutorial❗️

But before we do that, let's set the stage with a brief overview of Fine-Tuning.

Fine-Tuning: The Traditional approach

Fine-tuning is a well-established method in machine learning where a pre-trained model is further trained (or "fine-tuned") on a specific task. This approach leverages the general knowledge the model has learned during its initial training (often on a large and diverse dataset) and adapts it to a particular use case.

Here’s a typical representation of traditional Fine-Tuning 👇