huggingface-projects/Deep-RL-Course-Certification
Viewer • Updated • 1.96k • 1.73k • 20
AI & ML interests
None defined yet.
Recent Activity
View all activity
sergiopaniego
updated a
dataset about 3 hours ago
sergiopaniego
posted an update 3 days ago
Post
6,683
New post-training example: SFT on agent traces with TRL
→ A large model works as a coding agent in OpenCode, run through OpenEnv
→ The tasks it solves become the training data
→ A 2B model goes from 5.5% to 34.4% on held-out tasks
About 500 traces get most of the gain. Five times more add only 3 to 4 points, so SFT gets the agent working and RL has to do the rest
Full guide, with the models, traces and every rollout open:
https://github.com/huggingface/trl/tree/main/examples/sft_agent_traces
→ A large model works as a coding agent in OpenCode, run through OpenEnv
→ The tasks it solves become the training data
→ A 2B model goes from 5.5% to 34.4% on held-out tasks
About 500 traces get most of the gain. Five times more add only 3 to 4 points, so SFT gets the agent working and RL has to do the rest
Full guide, with the models, traces and every rollout open:
https://github.com/huggingface/trl/tree/main/examples/sft_agent_traces
- 1 reply
pcuenq
updated a
dataset 7 days ago
sergiopaniego
posted an update 10 days ago
Post
4,683
TRL now supports MOPD (Multi-Teacher On-Policy Distillation) natively❗️
Many frontier open models use it. labs train one RL expert per domain and then distill them all into one model
Blog with what it is, who uses it and an experiment you can run on HF Jobs with one command
https://huggingface.co/blog/sergiopaniego/mopd-trl
Many frontier open models use it. labs train one RL expert per domain and then distill them all into one model
Blog with what it is, who uses it and an experiment you can run on HF Jobs with one command
https://huggingface.co/blog/sergiopaniego/mopd-trl
coyotte508
posted an update 10 days ago
Post
197
Added a "ship it" button to my personal harness just to push the model to go ahead, push, commit or PR. Fun!
sergiopaniego
posted an update 14 days ago
Post
1,200
if you want to deep dive into how the new @MistralAI Large 4 was built, we sat down with the team to learn straight from them!
scaling, specialization, evals, cyber, sandboxing and why open weights matter
a really detailed look at what it takes to build and release an open frontier model, super recommended
https://www.youtube.com/watch?v=FczbSauVvjw
scaling, specialization, evals, cyber, sandboxing and why open weights matter
a really detailed look at what it takes to build and release an open frontier model, super recommended
https://www.youtube.com/watch?v=FczbSauVvjw
sergiopaniego
posted an update 15 days ago
Post
3,128
Mistral Large 4 (Le Chonk 🐈) is now live on the Hub and you can post-train it with TRL today!
SFT + LoRA example ready, runs on a single 8xB300 node
> model: mistralai/Mistral-Large-4-1T-A52B
> TRL example: https://github.com/huggingface/trl/tree/main/examples/sft_mistral_large_4
> blog: https://huggingface.co/blog/mistral-large4
SFT + LoRA example ready, runs on a single 8xB300 node
> model: mistralai/Mistral-Large-4-1T-A52B
> TRL example: https://github.com/huggingface/trl/tree/main/examples/sft_mistral_large_4
> blog: https://huggingface.co/blog/mistral-large4
AdinaY
submitted a
paper to Daily Papers 15 days ago
sergiopaniego
posted an update 16 days ago
Post
3,077
how do you post-train an agent?
we laid it out as a 5-step curriculum, from SFT to RL through a real harness, with livestreams, code and docs for each step
https://huggingface.co/blog/huggingface/train-agents
we laid it out as a 5-step curriculum, from SFT to RL through a real harness, with livestreams, code and docs for each step
https://huggingface.co/blog/huggingface/train-agents
- 2 replies
sergiopaniego
posted an update 17 days ago
Post
187
Last week we closed the Training Agents series IRL at the GitHub offices in SF
4 talks, a panel and audience Q&A. almost 2h on post-training agents with open models
> Learnings from enterprise RL (Govind Kamtamneni Microsoft)
> Environments and benchmarks in industry (Victor Barres Mercor)
> How to train your agents with Miles (Tao Lin RadixArk)
> Why we built OpenEnv Arena (Xiangyi Li BenchFlow)
Full recording + Slides
https://www.youtube.com/watch?v=CKFUx7COIwQ
4 talks, a panel and audience Q&A. almost 2h on post-training agents with open models
> Learnings from enterprise RL (Govind Kamtamneni Microsoft)
> Environments and benchmarks in industry (Victor Barres Mercor)
> How to train your agents with Miles (Tao Lin RadixArk)
> Why we built OpenEnv Arena (Xiangyi Li BenchFlow)
Full recording + Slides
https://www.youtube.com/watch?v=CKFUx7COIwQ
Post
387
tokenizers 0.23.3 is out!https://github.com/huggingface/tokenizers/releases/tag/v0.23.3
The release is python-only and fixes a few very annoying issues
sergiopaniego
posted an update about 1 month ago
Post
3,663
sharing the slides from my talk at Kernel Panic in Madrid
It's basically "The ultimate guide to multi-harness RL" turned slides and condensed into a 20 min talk
FineEnvs/multi-harness-rl-slides
congrats to @Helmcode for the organization!
blog: FineEnvs/multi-harness-rl
It's basically "The ultimate guide to multi-harness RL" turned slides and condensed into a 20 min talk
FineEnvs/multi-harness-rl-slides
congrats to @Helmcode for the organization!
blog: FineEnvs/multi-harness-rl
sergiopaniego
posted an update about 1 month ago
Post
343
Repo2RLEnv just shipped TaskSmith + 50 high quality RL envs generated from HF repos 🔨
TaskSmith is a specialized harness that turns a merged PR into a verified RL env
the envs come from HF repos (Transformers, TRL, PEFT, Accelerate, Diffusers), shipped as Harbor tasks you can eval or train on
> code: github.com/huggingface/Repo2RLEnv
> dataset: huggingface.co/datasets/FineEnvs/HF_ML_Tasksmith
TaskSmith is a specialized harness that turns a merged PR into a verified RL env
the envs come from HF repos (Transformers, TRL, PEFT, Accelerate, Diffusers), shipped as Harbor tasks you can eval or train on
> code: github.com/huggingface/Repo2RLEnv
> dataset: huggingface.co/datasets/FineEnvs/HF_ML_Tasksmith
sergiopaniego
posted an update about 1 month ago
Post
3,858
ThinkingBox from @microsoft is now available as an OpenEnv env (cc @tuhink 🤗 )!
> ThinkingBox is a sandbox for testing agents on business workflows. it simulates a customer, gives the agent MCP tools over a real database, and at the end checks what changed in that database instead of trusting the agent's last message
> ThinkingBox-Bench is the benchmark built on it: 507 tasks across retail, insurance, travel, banking and consulting
> the OpenEnv env runs each task as an episode in its own isolated backend and returns a pass/fail reward from those checks
https://huggingface.co/blog/microsoft/thinkingbox
> ThinkingBox is a sandbox for testing agents on business workflows. it simulates a customer, gives the agent MCP tools over a real database, and at the end checks what changed in that database instead of trusting the agent's last message
> ThinkingBox-Bench is the benchmark built on it: 507 tasks across retail, insurance, travel, banking and consulting
> the OpenEnv env runs each task as an episode in its own isolated backend and returns a pass/fail reward from those checks
https://huggingface.co/blog/microsoft/thinkingbox
sergiopaniego
posted an update about 1 month ago
Request to remove my entry from certified_users.csv
1
#4 opened about 1 month ago
by 
Deepuamrita23115
Delete row per user request
#5 opened about 1 month ago
by 
pcuenq
Post
333
I put together a Space to showcase tokenization-in-the-browser libs we maintain at HF
Let me know what you think!
sbrandeis/tokenizers-wasm-demo
Let me know what you think!
sbrandeis/tokenizers-wasm-demo
AdinaY
submitted a
paper to Daily Papers about 2 months ago
sergiopaniego
posted an update about 2 months ago
Post
481
ICYMI, Async GRPO in TRL now supports LoRA and we wrote a looong blog testing it
> the adapter is a few megabytes, so the weight sync is a file instead of an NCCL transfer
> 3 HF Jobs: 1 trainer and 2 vLLM replicas
> the adapter travels through an HF Storage Bucket mounted in all 3 at the same path
> a proxy in front of the replicas routes each rollout to the one already holding its KV prefix
https://huggingface.co/blog/asyncgrpo-lora-hfjobs
> the adapter is a few megabytes, so the weight sync is a file instead of an NCCL transfer
> 3 HF Jobs: 1 trainer and 2 vLLM replicas
> the adapter travels through an HF Storage Bucket mounted in all 3 at the same path
> a proxy in front of the replicas routes each rollout to the one already holding its KV prefix
https://huggingface.co/blog/asyncgrpo-lora-hfjobs