trl-internal-testing/tiny-Glm5NextForConditionalGeneration
Image-Text-to-Text • 10M • Updated • 222
Albert Villanova del Moral
albertvillanova
AI & ML interests
ML Engineer @ Hugging Face: Agents (Science)
Recent Activity
new activity about 12 hours ago
trl-internal-testing/tiny-Gemma2ForCausalLM:Upload Gemma2ForCausalLM new activity about 12 hours ago
trl-internal-testing/tiny-GemmaForCausalLM:Upload GemmaForCausalLM new activity about 12 hours ago
trl-internal-testing/tiny-Qwen2ForSequenceClassification-2.5:Upload Qwen2ForSequenceClassificationOrganizations
Upload Gemma2ForCausalLM
#4 opened about 13 hours ago
by 
albertvillanova
Upload GemmaForCausalLM
#4 opened about 14 hours ago
by albertvillanova
Upload Qwen2ForSequenceClassification
1
#2 opened about 14 hours ago
by albertvillanova
Upload Qwen2ForCausalLM
#3 opened about 14 hours ago
by albertvillanova
Upload MistralForCausalLM
#2 opened 4 days ago
by albertvillanova
Upload MistralForCausalLM
#2 opened 4 days ago
by albertvillanova
Upload LlamaForCausalLM
#2 opened 4 days ago
by albertvillanova
Upload LlamaForCausalLM
#2 opened 4 days ago
by albertvillanova
Upload LlamaForCausalLM
#3 opened 4 days ago
by albertvillanova
Upload Glm5NextForConditionalGeneration
#1 opened 4 days ago
by albertvillanova
updated a
model 6 days ago
published a
model 6 days ago
posted an update 12 days ago
Post
269
Day-0 Mistral Large 4 support in TRL 🚀
- LoRA SFT of the 1T-A52B model on one 8xB300 node, on top of the FP8 checkpoint
- Experts split 8 ways with expert parallelism, everything else sharded with FSDP, all set at load time
- Assistant-only loss out of the box with its training chat template
- ~32.7k tokens per step in ~8 s
Script + Slurm launcher 👇
https://github.com/huggingface/trl/tree/main/examples/sft_mistral_large_4
https://huggingface.co/posts/sergiopaniego/408182860646834
- LoRA SFT of the 1T-A52B model on one 8xB300 node, on top of the FP8 checkpoint
- Experts split 8 ways with expert parallelism, everything else sharded with FSDP, all set at load time
- Assistant-only loss out of the box with its training chat template
- ~32.7k tokens per step in ~8 s
Script + Slurm launcher 👇
https://github.com/huggingface/trl/tree/main/examples/sft_mistral_large_4
https://huggingface.co/posts/sergiopaniego/408182860646834
reacted to sergiopaniego's post with 🚀 12 days ago
Post
3,099
Mistral Large 4 (Le Chonk 🐈) is now live on the Hub and you can post-train it with TRL today!
SFT + LoRA example ready, runs on a single 8xB300 node
> model: mistralai/Mistral-Large-4-1T-A52B
> TRL example: https://github.com/huggingface/trl/tree/main/examples/sft_mistral_large_4
> blog: https://huggingface.co/blog/mistral-large4
SFT + LoRA example ready, runs on a single 8xB300 node
> model: mistralai/Mistral-Large-4-1T-A52B
> TRL example: https://github.com/huggingface/trl/tree/main/examples/sft_mistral_large_4
> blog: https://huggingface.co/blog/mistral-large4
Sync the chat template with the released checkpoint
#1 opened 12 days ago
by albertvillanova
published a
model 12 days ago
liked a
model 12 days ago
updated a
model 13 days ago
published a
model 13 days ago
Upload Lfm2VlForConditionalGeneration
#1 opened about 1 month ago
by albertvillanova