AmkyawDev-LLM-V3

Burmese Language Model Fine-tuning Project using LoRA/QLoRA with Unsloth

🇲🇲 AmkyawDev-LLM-V3

Burmese Language Model | Qwen2.5-1.5B | Unsloth Fine-tuned

Model: HuggingFace | Space: Gradio

Drive Progress With Intelligent Systems

---

📋 Table of Contents

• Project Structure
• Quick Start
• Model Details
• Training
• Deployment
• API Usage
• Limitations
• License
• Acknowledgments

---

📁 Project Structure

AmkyawDev-LLM-V3/
├── 📁 data/                  # Dataset ပိုင်း
│   ├── raw/                  # မပြုပြင်ရသေးသော data များ (Wiki, Social, Books)
│   ├── processed/            # Clean လုပ်ပြီးသား (Unicode normalized)
│   └── chat_format/          # ShareGPT သို့မဟုတ် Alpaca format
├── 📁 training/              # Training scripts များ
│   ├── config.yaml           # LoRA/QLoRA hyperparameters
│   ├── train_lora.py         # Standard PEFT training
│   ├── train_unsloth.py      # Unsloth memory-efficient training
│   └── requirements.txt      # Dependencies
├── 📁 model/                 # Output ပိုင်း
│   ├── adapter/              # Trained LoRA weights
│   └── merged/               # Base + LoRA merged version
├── 📁 deployment/            # API နှင့် UI ပိုင်း
│   ├── 📁 api/               # FastAPI သို့မဟုတ် LiteLLM Proxy
│   └── 📁 web_ui/            # Gradio Chat Interface
├── 📁 scripts/               # Utility scripts
│   ├── convert_to_unicode.py
│   ├── push_to_hub.py        # Push to HuggingFace Hub
│   └── push_space.py         # Push to HuggingFace Spaces
└── README.md

---

🚀 Quick Start

1. Install Dependencies

cd training
pip install -r requirements.txt

2. Prepare Data

# Convert raw data to normalized Unicode
python scripts/convert_to_unicode.py data/raw --output data/processed

3. Configure Training

Edit training/config.yaml:

model:
  name: "Qwen/Qwen2.5-1.5B-Instruct"

lora:
  r: 16
  lora_alpha: 32

training:
  num_train_epochs: 3
  learning_rate: 2e-4
  bf16: true

4. Train Model

# Unsloth (Recommended - Memory Efficient)
python training/train_unsloth.py

# Standard PEFT
python training/train_lora.py

5. Deploy

# Push to HuggingFace Hub
python scripts/push_to_hub.py

# Create Gradio Space
python scripts/push_space.py

---

📊 Model Details

Property	Value
Base Model	Qwen/Qwen2.5-1.5B-Instruct
Architecture	Transformer (Decoder-only)
Training Method	Unsloth + QLoRA (4-bit)
Context Length	2048 tokens
Parameters	1.5B
Fine-tuning Framework	TRL + PEFT

LoRA Configuration

lora:
  r: 16
  lora_alpha: 32
  lora_dropout: 0.05
  target_modules:
    - q_proj
    - k_proj
    - v_proj
    - o_proj
    - gate_proj
    - up_proj
    - down_proj

---

🔧 Training

Requirements

torch>=2.0.0
transformers>=4.36.0
unsloth>=2024.1.0
peft>=0.8.0
trl>=0.7.0
datasets>=2.14.0
accelerate>=0.25.0

Data Format (ShareGPT)

{
  "messages": [
    {"role": "system", "content": "You are a helpful assistant."},
    {"role": "user", "content": "မြန်မာစကားအကြောင်းပါ"},
    {"role": "assistant", "content": "ဟုတ်ပါတယ်။"}
  ]
}

Data Format (Alpaca)

{
  "prompt": "မြန်မာစကားသင်ပါ",
  "response": "ဟုတ်ပါတယ်။"
}

---

🌐 Deployment

HuggingFace Space (Live Demo)

🔗 URL: https://huggingface.co/spaces/amkyawdev/AmkyawDev-LLM-V3

Features:

• 🖥️ Web UI Chat Interface
• ⚙️ Adjustable Parameters (temperature, max_tokens)
• 📱 Mobile-friendly

Local Deployment

pip install gradio transformers peft
python deployment/web_ui/app.py

---

📡 API Usage

FastAPI

from fastapi import FastAPI
from transformers import AutoModelForCausalLM, AutoTokenizer
import torch

app = FastAPI()

model = AutoModelForCausalLM.from_pretrained("Qwen/Qwen2.5-1.5B-Instruct")
tokenizer = AutoTokenizer.from_pretrained("amkyawdev/AmkyawDev-LLM-V3")

@app.post("/generate")
def generate(prompt: str):
    inputs = tokenizer(prompt, return_tensors="pt").to("cuda")
    outputs = model.generate(**inputs, max_new_tokens=256)
    return {"response": tokenizer.decode(outputs[0])}

---

⚠️ Limitations

1. Knowledge Cutoff - သတင်းအချက်လဒ်များသည် training data ပါချိန်အထိပါတယ်။
2. Language Bias - မြန်မာဘာသာစကားအတွက် အပြည့်အစုံ မဟုတ်ပါတယ်။
3. Hallucination - တစ်ခါတစ်ရာ မှားယွင်းတဲ့ အရာများပါတယ်။

---

📝 License

MIT License

Copyright (c) 2024 Amkyaw AI

---

🙏 Acknowledgments

• Qwen Team - Base model
• Unsloth AI - Memory efficient training
• Hugging Face - Infrastructure
• TRL - SFTTrainer
• PEFT - LoRA implementation

---

🤝 Connect

• GitHub
• HuggingFace

Amkyaw AI - Drive Progress With Intelligent Systems