agents-course/certificates
Updated • 42.3k • 104
AI & ML interests
None defined yet.
Recent Activity
View all activity
sergiopaniego
updated a
dataset about 8 hours ago
burtenshaw
updated a
dataset about 9 hours ago
sergiopaniego
updated 2
datasets about 10 hours ago
Jofthomas
updated a
dataset about 10 hours ago
sergiopaniego
posted an update 2 days ago
Post
5,334
New post-training example: SFT on agent traces with TRL
→ A large model works as a coding agent in OpenCode, run through OpenEnv
→ The tasks it solves become the training data
→ A 2B model goes from 5.5% to 34.4% on held-out tasks
About 500 traces get most of the gain. Five times more add only 3 to 4 points, so SFT gets the agent working and RL has to do the rest
Full guide, with the models, traces and every rollout open:
https://github.com/huggingface/trl/tree/main/examples/sft_agent_traces
→ A large model works as a coding agent in OpenCode, run through OpenEnv
→ The tasks it solves become the training data
→ A 2B model goes from 5.5% to 34.4% on held-out tasks
About 500 traces get most of the gain. Five times more add only 3 to 4 points, so SFT gets the agent working and RL has to do the rest
Full guide, with the models, traces and every rollout open:
https://github.com/huggingface/trl/tree/main/examples/sft_agent_traces
- 1 reply
sergiopaniego
posted an update 9 days ago
Post
4,667
TRL now supports MOPD (Multi-Teacher On-Policy Distillation) natively❗️
Many frontier open models use it. labs train one RL expert per domain and then distill them all into one model
Blog with what it is, who uses it and an experiment you can run on HF Jobs with one command
https://huggingface.co/blog/sergiopaniego/mopd-trl
Many frontier open models use it. labs train one RL expert per domain and then distill them all into one model
Blog with what it is, who uses it and an experiment you can run on HF Jobs with one command
https://huggingface.co/blog/sergiopaniego/mopd-trl
sergiopaniego
posted an update 13 days ago
Post
1,190
if you want to deep dive into how the new @MistralAI Large 4 was built, we sat down with the team to learn straight from them!
scaling, specialization, evals, cyber, sandboxing and why open weights matter
a really detailed look at what it takes to build and release an open frontier model, super recommended
https://www.youtube.com/watch?v=FczbSauVvjw
scaling, specialization, evals, cyber, sandboxing and why open weights matter
a really detailed look at what it takes to build and release an open frontier model, super recommended
https://www.youtube.com/watch?v=FczbSauVvjw
sergiopaniego
posted an update 14 days ago
Post
3,119
Mistral Large 4 (Le Chonk 🐈) is now live on the Hub and you can post-train it with TRL today!
SFT + LoRA example ready, runs on a single 8xB300 node
> model: mistralai/Mistral-Large-4-1T-A52B
> TRL example: https://github.com/huggingface/trl/tree/main/examples/sft_mistral_large_4
> blog: https://huggingface.co/blog/mistral-large4
SFT + LoRA example ready, runs on a single 8xB300 node
> model: mistralai/Mistral-Large-4-1T-A52B
> TRL example: https://github.com/huggingface/trl/tree/main/examples/sft_mistral_large_4
> blog: https://huggingface.co/blog/mistral-large4
sergiopaniego
posted an update 15 days ago
Post
3,071
how do you post-train an agent?
we laid it out as a 5-step curriculum, from SFT to RL through a real harness, with livestreams, code and docs for each step
https://huggingface.co/blog/huggingface/train-agents
we laid it out as a 5-step curriculum, from SFT to RL through a real harness, with livestreams, code and docs for each step
https://huggingface.co/blog/huggingface/train-agents
- 2 replies
sergiopaniego
posted an update 16 days ago
Post
182
Last week we closed the Training Agents series IRL at the GitHub offices in SF
4 talks, a panel and audience Q&A. almost 2h on post-training agents with open models
> Learnings from enterprise RL (Govind Kamtamneni Microsoft)
> Environments and benchmarks in industry (Victor Barres Mercor)
> How to train your agents with Miles (Tao Lin RadixArk)
> Why we built OpenEnv Arena (Xiangyi Li BenchFlow)
Full recording + Slides
https://www.youtube.com/watch?v=CKFUx7COIwQ
4 talks, a panel and audience Q&A. almost 2h on post-training agents with open models
> Learnings from enterprise RL (Govind Kamtamneni Microsoft)
> Environments and benchmarks in industry (Victor Barres Mercor)
> How to train your agents with Miles (Tao Lin RadixArk)
> Why we built OpenEnv Arena (Xiangyi Li BenchFlow)
Full recording + Slides
https://www.youtube.com/watch?v=CKFUx7COIwQ
sergiopaniego
posted an update about 1 month ago
Post
3,660
sharing the slides from my talk at Kernel Panic in Madrid
It's basically "The ultimate guide to multi-harness RL" turned slides and condensed into a 20 min talk
FineEnvs/multi-harness-rl-slides
congrats to @Helmcode for the organization!
blog: FineEnvs/multi-harness-rl
It's basically "The ultimate guide to multi-harness RL" turned slides and condensed into a 20 min talk
FineEnvs/multi-harness-rl-slides
congrats to @Helmcode for the organization!
blog: FineEnvs/multi-harness-rl
sergiopaniego
posted an update about 1 month ago
Post
340
Repo2RLEnv just shipped TaskSmith + 50 high quality RL envs generated from HF repos 🔨
TaskSmith is a specialized harness that turns a merged PR into a verified RL env
the envs come from HF repos (Transformers, TRL, PEFT, Accelerate, Diffusers), shipped as Harbor tasks you can eval or train on
> code: github.com/huggingface/Repo2RLEnv
> dataset: huggingface.co/datasets/FineEnvs/HF_ML_Tasksmith
TaskSmith is a specialized harness that turns a merged PR into a verified RL env
the envs come from HF repos (Transformers, TRL, PEFT, Accelerate, Diffusers), shipped as Harbor tasks you can eval or train on
> code: github.com/huggingface/Repo2RLEnv
> dataset: huggingface.co/datasets/FineEnvs/HF_ML_Tasksmith
sergiopaniego
posted an update about 1 month ago
Post
3,855
ThinkingBox from @microsoft is now available as an OpenEnv env (cc @tuhink 🤗 )!
> ThinkingBox is a sandbox for testing agents on business workflows. it simulates a customer, gives the agent MCP tools over a real database, and at the end checks what changed in that database instead of trusting the agent's last message
> ThinkingBox-Bench is the benchmark built on it: 507 tasks across retail, insurance, travel, banking and consulting
> the OpenEnv env runs each task as an episode in its own isolated backend and returns a pass/fail reward from those checks
https://huggingface.co/blog/microsoft/thinkingbox
> ThinkingBox is a sandbox for testing agents on business workflows. it simulates a customer, gives the agent MCP tools over a real database, and at the end checks what changed in that database instead of trusting the agent's last message
> ThinkingBox-Bench is the benchmark built on it: 507 tasks across retail, insurance, travel, banking and consulting
> the OpenEnv env runs each task as an episode in its own isolated backend and returns a pass/fail reward from those checks
https://huggingface.co/blog/microsoft/thinkingbox
sergiopaniego
posted an update about 1 month ago
sergiopaniego
posted an update about 2 months ago
Post
480
ICYMI, Async GRPO in TRL now supports LoRA and we wrote a looong blog testing it
> the adapter is a few megabytes, so the weight sync is a file instead of an NCCL transfer
> 3 HF Jobs: 1 trainer and 2 vLLM replicas
> the adapter travels through an HF Storage Bucket mounted in all 3 at the same path
> a proxy in front of the replicas routes each rollout to the one already holding its KV prefix
https://huggingface.co/blog/asyncgrpo-lora-hfjobs
> the adapter is a few megabytes, so the weight sync is a file instead of an NCCL transfer
> 3 HF Jobs: 1 trainer and 2 vLLM replicas
> the adapter travels through an HF Storage Bucket mounted in all 3 at the same path
> a proxy in front of the replicas routes each rollout to the one already holding its KV prefix
https://huggingface.co/blog/asyncgrpo-lora-hfjobs
sergiopaniego
posted an update 2 months ago
Post
3,413
while preparing the last class of the Training Agents live series during the summer, i spent some time reading the post-training sections of many frontier model reports, to learn how they use RL environments to improve their models, and wrote a blog about it
if you use any kind of coding harness, or you saw the Blender scenes that went viral recently, this might be interesting to you
Blog: https://huggingface.co/blog/sergiopaniego/rl-environments-2026
if you use any kind of coding harness, or you saw the Blender scenes that went viral recently, this might be interesting to you
Blog: https://huggingface.co/blog/sergiopaniego/rl-environments-2026
sergiopaniego
posted an update 2 months ago
Post
2,275
Can you do RL over taste?
I've spent some time reproducing, in the open, Surya N's idea of training a model to paint with code. It's a coding model that learns to paint watercolours by writing JS code, trained with GRPO. I used TRL and OpenEnv for this, with the whole pipeline running on Hugging Face.
The interesting part is that the reward has no correct answer, unlike a math problem. In this case it's based on the artistic preferences of the person who builds the dataset.
Everything is published: the environment, the reference pool, the trained adapters, every painting of every run with the code that made it, and a write-up with all the decisions, including the ones that went wrong.
Blog post: https://huggingface.co/blog/train-to-paint-with-code
I've spent some time reproducing, in the open, Surya N's idea of training a model to paint with code. It's a coding model that learns to paint watercolours by writing JS code, trained with GRPO. I used TRL and OpenEnv for this, with the whole pipeline running on Hugging Face.
The interesting part is that the reward has no correct answer, unlike a math problem. In this case it's based on the artistic preferences of the person who builds the dataset.
Everything is published: the environment, the reference pool, the trained adapters, every painting of every run with the code that made it, and a write-up with all the decisions, including the ones that went wrong.
Blog post: https://huggingface.co/blog/train-to-paint-with-code
sergiopaniego
posted an update 3 months ago
Post
627
catching up on some bookmarked reads from the summer, reading Antidoom from @liquidai
small reasoning models get stuck more easily when the task involves a long thinking trace and a hard problem. It starts repeating the same word over and over again ("Wait", "Alternatively"…), each repetition makes the next one likelier, and the generation is spent before it reaches an answer
they measured it, 10.2% of completions for an early LFM2.5-2.6B checkpoint and 22.9% for Qwen3.5-4B at greedy. After training those drop to 1.4% and 1.0%
the fix is FTPO (final token preference optimization). What I like is how narrow it is, it only touches the single token where the loop starts
three ways it differs from DPO:
> trains one token position, mid-generation, instead of whole sequences
> spreads probability across ~20 plausible alternatives instead of swapping one overtrained token for another
> keeps the regularizer in logit space, no softmax, so the rest of the vocabulary stays put
the third one is what makes it usable. If you want to edit one position without disturbing the model, you can't have a loss that reshuffles the other 150k logits on the way
and their explanation abt the result: the training teaches the model nothing new about math or code, it clears the failure mode that was blocking answers the model could already produce
full blog > https://www.liquid.ai/blog/antidoom
FTPO itself comes from Antislop, where it was built to strip overused phrasing. LiquidAI retargeted it to doom loops
and under the hood it's a subclass of TRL's DPOTrainer with compute_loss overridden, around 90 lines of loss and no new trainer
we documented that pattern in TRL's docs
https://huggingface.co/docs/trl/main/en/customization#change-the-training-objective
small reasoning models get stuck more easily when the task involves a long thinking trace and a hard problem. It starts repeating the same word over and over again ("Wait", "Alternatively"…), each repetition makes the next one likelier, and the generation is spent before it reaches an answer
they measured it, 10.2% of completions for an early LFM2.5-2.6B checkpoint and 22.9% for Qwen3.5-4B at greedy. After training those drop to 1.4% and 1.0%
the fix is FTPO (final token preference optimization). What I like is how narrow it is, it only touches the single token where the loop starts
three ways it differs from DPO:
> trains one token position, mid-generation, instead of whole sequences
> spreads probability across ~20 plausible alternatives instead of swapping one overtrained token for another
> keeps the regularizer in logit space, no softmax, so the rest of the vocabulary stays put
the third one is what makes it usable. If you want to edit one position without disturbing the model, you can't have a loss that reshuffles the other 150k logits on the way
and their explanation abt the result: the training teaches the model nothing new about math or code, it clears the failure mode that was blocking answers the model could already produce
full blog > https://www.liquid.ai/blog/antidoom
FTPO itself comes from Antislop, where it was built to strip overused phrasing. LiquidAI retargeted it to doom loops
and under the hood it's a subclass of TRL's DPOTrainer with compute_loss overridden, around 90 lines of loss and no new trainer
we documented that pattern in TRL's docs
https://huggingface.co/docs/trl/main/en/customization#change-the-training-objective
- 1 reply
sergiopaniego
posted an update 3 months ago
Post
429
super interesting new paper from Microsoft "Agent Lightning v1.0: Towards Harnessed Agentic RL" by Zhiyuan He et al.
same idea we've seen already several times: you train the agent inside the real harness it ships with, instead of a reimplementation of it
now that recipe has a name → harnessed agentic RL
paper: huggingface.co/papers/2608.17528
the tricky bit they nail down: one rollout is not one training sample
the harness calls the model many times, so a single episode → a variable number of (prompt, response) rows
you don't even know the batch size until the episode finishes running
its real contribution is being first to systematically map the four problems that fall out of that:
> retokenization + sample merging
> advantage calculation over a variable sample count
> loss normalization at the rollout level, not per sample
> backend scheduling when the batch size is dynamic
and it actually works → plain RL inside the real harness, no reimplementation
Qwen3.5-9B on SWE-bench Verified 41.8 → 56.4 (+14.6), with only ~6k examples
the whole thing is ~3,500 lines, any harness, self-hosted k8s
from our side, we've shared some materials on the same line you may want to check out :)
> Agentic RL: Token-In, Token-Out Done Right: https://huggingface.co/blog/huggingface/tito
> a full worked example, opencode owning its loop trained with GRPO: https://huggingface.co/blog/sergiopaniego/trl-openenv-harness-training
> Harness, Scaffold, and the AI Agent Terms Worth Getting Right: https://huggingface.co/blog/agent-glossary
on a similar line:
https://x.com/SergioPaniego/status/2062911580564496576
same idea we've seen already several times: you train the agent inside the real harness it ships with, instead of a reimplementation of it
now that recipe has a name → harnessed agentic RL
paper: huggingface.co/papers/2608.17528
the tricky bit they nail down: one rollout is not one training sample
the harness calls the model many times, so a single episode → a variable number of (prompt, response) rows
you don't even know the batch size until the episode finishes running
its real contribution is being first to systematically map the four problems that fall out of that:
> retokenization + sample merging
> advantage calculation over a variable sample count
> loss normalization at the rollout level, not per sample
> backend scheduling when the batch size is dynamic
and it actually works → plain RL inside the real harness, no reimplementation
Qwen3.5-9B on SWE-bench Verified 41.8 → 56.4 (+14.6), with only ~6k examples
the whole thing is ~3,500 lines, any harness, self-hosted k8s
from our side, we've shared some materials on the same line you may want to check out :)
> Agentic RL: Token-In, Token-Out Done Right: https://huggingface.co/blog/huggingface/tito
> a full worked example, opencode owning its loop trained with GRPO: https://huggingface.co/blog/sergiopaniego/trl-openenv-harness-training
> Harness, Scaffold, and the AI Agent Terms Worth Getting Right: https://huggingface.co/blog/agent-glossary
on a similar line:
https://x.com/SergioPaniego/status/2062911580564496576