Abhay557/gemma-mini-code-agent

🧠 Mini Coding Agent - Fine-tuned Gemma-3-1B-IT

A small coding assistant (~1B parameters) built by fine-tuning Gemma-3-1B-IT on coding instruction datasets. Think of it as a tiny Claude Code you can run on a free Google Colab T4 GPU.

Model Details

Property	Value
Base Model	google/gemma-3-1b-it
Parameters	~1B (actual ~1.3B)
Training Method	LoRA (Low-Rank Adaptation) + 4-bit Quantization
Trainable Parameters	~1.5% of total
Dataset	ise-uiuc/Magicoder-OSS-Instruct-75K or nvidia/OpenCodeInstruct
VRAM Usage	~6-10GB peak (fits on Colab T4)
Training Time	~30-60 min for 50K samples, 2 epochs

Why These Choices?

• Gemma-3-1B-IT: The smallest official Gemma model. Already instruction-tuned, so it understands chat format.
• LoRA: Only trains adapter layers (~20M params), keeping VRAM low while still learning coding patterns.
• 4-bit (NF4) Quantization: Cuts memory by ~4x with minimal quality loss.
• Magicoder Dataset: Proven recipe (arxiv:2312.02120) using real open-source code snippets as seeds — better than raw code pairs.
• OpenCodeInstruct: Higher quality synthetic data with unit tests (arxiv:2504.04030). Use a subset for Colab.

Quick Start in Google Colab

Step 1: Setup

!pip install -q transformers trl peft datasets accelerate bitsandbytes huggingface_hub

Step 2: Authenticate

from huggingface_hub import notebook_login
notebook_login()

IMPORTANT: Visit https://huggingface.co/google/gemma-3-1b-it and ACCEPT the license before training!

Step 3: Change Runtime to GPU

Go to Runtime > Change runtime type > T4 GPU

Step 4: Run Training

Download and run train_colab.py:

# In a Colab cell:
!wget https://huggingface.co/Abhay557/gemma-mini-code-agent/raw/main/train_colab.py
!python train_colab.py

Or copy-paste the contents of train_colab.py directly into a Colab cell.

Step 5: Chat with your Agent

After training, use the built-in chat_with_agent() function from the script, or download inference.py:

!wget https://huggingface.co/Abhay557/gemma-mini-code-agent/raw/main/inference.py
!python inference.py

Configurable Parameters

Edit these in train_colab.py before running:

Param	Default	Description
MAX_SAMPLES	50000	Dataset subset size (reduce for faster runs)
NUM_EPOCHS	2	Training epochs
LEARNING_RATE	5e-5	LoRA learning rate
LORA_R	16	LoRA rank
LORA_ALPHA	32	LoRA scaling
MAX_SEQ_LENGTH	1024	Max tokens per sequence
GRAD_ACCUM	16	Gradient accumulation steps

Datasets

Dataset	Size	Best For	Paper
ise-uiuc/Magicoder-OSS-Instruct-75K	75K	Quick experiments, proven recipe	arxiv:2312.02120
nvidia/OpenCodeInstruct	5M	Best quality, use subset for Colab	arxiv:2504.04030

To switch datasets, change the DATASET_NAME variable in the script.

Expected Results

This won't match Claude Code (that's ~100B+ params), but it can:

• ✅ Write small Python functions
• ✅ Explain algorithms
• ✅ Debug simple code
• ✅ Answer basic coding interview questions

Benchmarks on similar 1B models fine-tuned with these datasets:

• HumanEval: ~30-40% pass@1 (base model: ~10-15%)
• MBPP: ~35-45% pass@1

Pushing to Hugging Face Hub

After training, uncomment these lines in the script:

# merged_model.push_to_hub("YOUR_USERNAME/gemma-3-1b-code-agent")
# tokenizer.push_to_hub("YOUR_USERNAME/gemma-3-1b-code-agent")

Troubleshooting

Issue	Fix
OOM error	Reduce MAX_SEQ_LENGTH to 512 or MAX_SAMPLES to 10000
Training too slow	Reduce MAX_SAMPLES to 10000, reduce NUM_EPOCHS to 1
Gemma license error	Visit the model page and click "Accept"
prepare_model_for_kbit_training import error	Make sure peft is up to date: !pip install -U peft

Architecture

Base: google/gemma-3-1b-it (Gemma3ForCausalLM)
  ├── 26 layers
  ├── 1152 hidden size
  ├── 4 attention heads
  └── 262k vocab

+ LoRA adapters (r=16, alpha=32)
  ├── q_proj, k_proj, v_proj, o_proj
  ├── gate_proj, up_proj, down_proj
  └── ~20M trainable params

+ 4-bit NF4 quantization
  └── ~3.5GB model footprint

License

• Base model: Gemma License
• This fine-tune: MIT
• Datasets: Check respective dataset pages

Citation

If you use this, cite the base papers:

@article{gemma3_2025,
  title={Gemma 3 Technical Report},
  author={Google DeepMind},
  year={2025}
}

@article{magicoder_2024,
  title={Magicoder: Source Code is All You Need},
  author={Wei, Yuxiang and others},
  journal={arXiv:2312.02120},
  year={2024}
}