Tiny-LLM CLI SFT (54M)
A 54 million parameter language model fine-tuned for CLI command generation.
Model Description
This model is a Supervised Fine-Tuned (SFT) version of jonmabe/tiny-llm-54m, trained to generate Unix/Linux shell commands from natural language instructions.
Training Data
- Geddy's NL2Bash dataset: ~2,300 natural language to bash command pairs
- NL2Bash benchmark: Standard benchmark for command translation
- Synthetic examples: Additional generated pairs
- Total: ~13,000 training pairs
Training Details
| Parameter | Value |
|---|---|
| Base Model | tiny-llm-54m |
| Training Steps | 2,000 |
| Best Checkpoint | Step 1,000 |
| Best Val Loss | 1.2456 |
| Learning Rate | 5e-5 |
| Batch Size | 16 |
| Hardware | NVIDIA RTX 5090 |
| Training Time | ~9 minutes |
Architecture
- Parameters: 54.93M
- Layers: 12
- Hidden Size: 512
- Attention Heads: 8
- Intermediate Size: 1408
- Max Position: 512
- Vocabulary: 32,000 tokens
- Features: RoPE, RMSNorm, SwiGLU, Weight Tying
Usage
Prompt Format
Instruction: <natural language description>
Command:
Example
from model import TinyLLM
import torch
# Load model
checkpoint = torch.load("best_model.pt", map_location="cpu")
model = TinyLLM(checkpoint["config"]["model"])
model.load_state_dict(checkpoint["model_state_dict"])
model.eval()
# Generate
prompt = "Instruction: Find all Python files modified in the last day\nCommand:"
# ... tokenize and generate
Limitations
⚠️ Known Issues:
- Tokenizer decode shows raw BPE tokens (Ġ = space, Ċ = newline)
- Model generates fragments of correct commands but output can be noisy
- Needs more training steps for reliable generation
- Small model size limits command complexity
Improvement Plan
- Fix tokenizer decode - Proper BPE to text conversion
- Longer training - 5,000-10,000 steps
- Data quality - Curate cleaner training pairs
- Lower LR - More stable convergence with 1e-5
License
Apache 2.0
Citation
@misc{tiny-llm-cli-sft-2026,
author = {Jon Mabe},
title = {Tiny-LLM CLI SFT: Small Language Model for Command Generation},
year = {2026},
publisher = {HuggingFace},
url = {https://huggingface.co/jonmabe/tiny-llm-cli-sft}
}