blackshell69/Qwen3.5-9B-distilled-coding-agentic

Qwen3.5-9B — distilled for coding + agentic tool-use

LoRA adapter for Qwen/Qwen3.5-9B, trained via sequence-level knowledge distillation from Qwen3.8-27B (teacher, served locally via llama.cpp) to strengthen coding and agentic (tool-calling) capability in a model small enough to run on consumer GPUs.

Training details

• Method: QLoRA — 4-bit NF4 base (frozen), LoRA rank 32 on all linear projections (q/k/v/o_proj, gate/up/down_proj), bf16 compute.
• Data: 9,640 samples of teacher-generated completions.
  • Coding — prompts from ise-uiuc/Magicoder-OSS-Instruct-75K; reference solutions discarded, the teacher writes its own.
  • Agentic tool-use — multi-turn conversations from glaiveai/glaive-function-calling-v2; every assistant turn (tool calls and post-tool-response answers alike) is regenerated by the teacher conditioned on the recorded history, in OpenAI tools/tool_calls format.
• Schedule: 2 epochs, cosine LR decay, completion-only loss (loss masked to the teacher's tokens via the student's own chat template).
• Result: final training loss 0.176, mean token accuracy 94.5%.
• Hardware: single RTX 5060 Ti (16GB), Windows, TRL + PEFT + bitsandbytes (SDPA attention).

Usage

🤗 PEFT

from transformers import AutoModelForCausalLM, AutoTokenizer
from peft import PeftModel

base = AutoModelForCausalLM.from_pretrained("Qwen/Qwen3.5-9B", dtype="bfloat16")
model = PeftModel.from_pretrained(base, "blackshell69/Qwen3.5-9B-distilled-coding-agentic")
tokenizer = AutoTokenizer.from_pretrained("Qwen/Qwen3.5-9B")

llama.cpp (GGUF)

Two ways to run this with llama.cpp:

Single merged file (Qwen3.5-9B-distilled-IQ4_NL.gguf, recommended):

llama-server -m Qwen3.5-9B-distilled-IQ4_NL.gguf --jinja

Base + LoRA adapter (Qwen3.5-9B-distilled-LoRA-F16.gguf), if you'd rather keep the adapter separate from a base GGUF you already have — numerically equivalent to the merged file:

llama-server -m Qwen3.5-9B-<any-quant>.gguf \
  --lora Qwen3.5-9B-distilled-LoRA-F16.gguf --jinja