Post
166
O NanoMathDataset é um dataset de quase 4M linhas de contas de matemática básica, veja ele em: pedrodev2026/NanoMathDataset
AxionLab
AxionLab-official
AI & ML interests
Owner of SupraLabs and iGPU Lover
Recent Activity
liked a model about 1 hour ago
SupraLabs/Supra-Title-Flan-85M upvoted a collection about 7 hours ago
Synthetic CoT published a model about 7 hours ago
SupraLabs/Supra-Title-Flan-85MOrganizations
reacted to pedrodev2026's post with 👍 about 10 hours ago
replied to their post 9 days ago
It turns out that you can cherry pick results, post an image, and act like it generalizes
wdym lmao, i never said the model generalizes it, i just posted a response about it when i tried it, the model isn't perfect, but isn't cool that a 50M params model quantized in 2Bit answer you like that?
replied to their post 9 days ago
3 days on a single RTX 5060 Ti 16GB
posted an update 10 days ago
Post
10821
THIS IS CRAZY! THE MODEL ON THE IMAGE(Supra-50M-Reasoning) answered correctly and its QUANTIZED IN 2BIT! THE RESPONSE IS CORRECT, IN A 15MB SIZE FILE!
- 14 replies
posted an update 12 days ago
Post
6572
We're happy to announce that we released a Reasoning tuned version of Supra-50M!
SupraLabs/Supra-50M-Reasoning
SupraLabs/Supra-50M-Reasoning
reacted to their post with 🚀 22 days ago
Post
267
Someone ran Supra-50M-Instruct ON A 1GHZ 1999 CPU
https://www.reddit.com/r/LocalLLM/comments/1tm21ar/i_see_your_strix_halo_and_raise_you_a_vintage/
"As a fun experiment, I decided to try running the recently released Supra-50m on a 26-year-old machine I keep for retro Windows 9.X games. Although the model was somewhat silly and inconsistent, the performance wasn't bad, reaching around 1.3 tok/s with CPU inference alone.
Since this CPU doesn't have SSE2, I changed from llama.cpp to llama2.ce and asked Claude to write a custom tokenizer.
It's crazy to think that with the right file size of 200 MB, we could have experienced this magic back in 1999" - u/drone_stonks, r/localllm
https://www.reddit.com/r/LocalLLM/comments/1tm21ar/i_see_your_strix_halo_and_raise_you_a_vintage/
"As a fun experiment, I decided to try running the recently released Supra-50m on a 26-year-old machine I keep for retro Windows 9.X games. Although the model was somewhat silly and inconsistent, the performance wasn't bad, reaching around 1.3 tok/s with CPU inference alone.
Since this CPU doesn't have SSE2, I changed from llama.cpp to llama2.ce and asked Claude to write a custom tokenizer.
It's crazy to think that with the right file size of 200 MB, we could have experienced this magic back in 1999" - u/drone_stonks, r/localllm
posted an update 22 days ago
Post
267
Someone ran Supra-50M-Instruct ON A 1GHZ 1999 CPU
https://www.reddit.com/r/LocalLLM/comments/1tm21ar/i_see_your_strix_halo_and_raise_you_a_vintage/
"As a fun experiment, I decided to try running the recently released Supra-50m on a 26-year-old machine I keep for retro Windows 9.X games. Although the model was somewhat silly and inconsistent, the performance wasn't bad, reaching around 1.3 tok/s with CPU inference alone.
Since this CPU doesn't have SSE2, I changed from llama.cpp to llama2.ce and asked Claude to write a custom tokenizer.
It's crazy to think that with the right file size of 200 MB, we could have experienced this magic back in 1999" - u/drone_stonks, r/localllm
https://www.reddit.com/r/LocalLLM/comments/1tm21ar/i_see_your_strix_halo_and_raise_you_a_vintage/
"As a fun experiment, I decided to try running the recently released Supra-50m on a 26-year-old machine I keep for retro Windows 9.X games. Although the model was somewhat silly and inconsistent, the performance wasn't bad, reaching around 1.3 tok/s with CPU inference alone.
Since this CPU doesn't have SSE2, I changed from llama.cpp to llama2.ce and asked Claude to write a custom tokenizer.
It's crazy to think that with the right file size of 200 MB, we could have experienced this magic back in 1999" - u/drone_stonks, r/localllm
posted an update 25 days ago
Post
242
We RELEASED!
SupraLabs just released our 50M model!
Base, Instruct Weights are there, you can use!
You can check blog to more informations!(Writing blog yet!)
SupraLabs just released our 50M model!
Base, Instruct Weights are there, you can use!
You can check blog to more informations!(Writing blog yet!)
- 2 replies
reacted to danielhanchen's post with 🚀🔥 about 1 month ago
Post
5853
We’re excited to announce that Unsloth has joined the PyTorch Ecosystem! 🔥🦥
Unsloth is an open-source project that makes training & running models more accurate and faster with less compute. Our mission is to make local AI accessible to everyone. Thanks to all of you for making this possible! 💕
Blog: https://unsloth.ai/blog/pytorch
GitHub: https://github.com/unslothai/unsloth
Unsloth is an open-source project that makes training & running models more accurate and faster with less compute. Our mission is to make local AI accessible to everyone. Thanks to all of you for making this possible! 💕
Blog: https://unsloth.ai/blog/pytorch
GitHub: https://github.com/unslothai/unsloth
- 2 replies
reacted to danielhanchen's post with 🚀🔥 about 1 month ago
Post
7733
We collaborated with NVIDIA to teach you how we made LLM training ~25% faster! 🚀
Learn how 3 optimizations help your home GPU train models faster:
1. Packed-sequence metadata caching
2. Double-buffered checkpoint reloads
3. Faster MoE routing
Guide: https://unsloth.ai/blog/nvidia-collab
GitHub: https://github.com/unslothai/unsloth
Learn how 3 optimizations help your home GPU train models faster:
1. Packed-sequence metadata caching
2. Double-buffered checkpoint reloads
3. Faster MoE routing
Guide: https://unsloth.ai/blog/nvidia-collab
GitHub: https://github.com/unslothai/unsloth
reacted to Crownelius's post with 🤯 about 1 month ago
Post
4213
Day 4-6 [05/05/2026]
Howdy,
Is anybody else willing to put a second mortgage on their house, just to spend 40k USD in compute credits? Just me? k...
I got dreams, man. The datasets I could build with 40k would be insane.
Somebody called me a genius the other day, they'd be shocked to find out, that I would put my house on the line for 30 days of runpod usage.
What would you do with it?
I would turn arxiv into a dataset. Turn each arxiv paper into a QnA.
Or... maybe if I got 40k USD in credit's Id end up like those 16 lost scientists.
Food for thought.
Anyways, I think I'm going to make a post once a week.
In the meantime you can find me building small llm's in discord here:
https://discord.gg/4DdwS9D8x9
edit:
So to be clear, I will not actually do this. But think of it this way. If you could pivot an entire industry with 10-30k would you do it?
Howdy,
Is anybody else willing to put a second mortgage on their house, just to spend 40k USD in compute credits? Just me? k...
I got dreams, man. The datasets I could build with 40k would be insane.
Somebody called me a genius the other day, they'd be shocked to find out, that I would put my house on the line for 30 days of runpod usage.
What would you do with it?
I would turn arxiv into a dataset. Turn each arxiv paper into a QnA.
Or... maybe if I got 40k USD in credit's Id end up like those 16 lost scientists.
Food for thought.
Anyways, I think I'm going to make a post once a week.
In the meantime you can find me building small llm's in discord here:
https://discord.gg/4DdwS9D8x9
edit:
So to be clear, I will not actually do this. But think of it this way. If you could pivot an entire industry with 10-30k would you do it?
- 7 replies
reacted to Crownelius's post with 🚀 about 1 month ago
Post
5313
Day 3 - 05/02/2026
Scamp ships, hits the wall. New plan...
Scamp came back from training today... Didn't go so well, I'm still unsure...
Fast benchmark, temperature 0.7, top_p 0.9:
- "Capital of France is" produced "covered by the Crown" (grammatical, factually wrong)
- "23 + 19 = ?" produced "23. Answer: 23. Answer: 23..." (loops, math broken)
- "def fibonacci(n):" produced a list of letters
It speaks English. It can't reason. At 8K vocab and 50M params, it was never going to.
Next build: 412M MoE-3E. Three experts (math, language, code), top-1 routing, random init, let specialization emerge from gradient signal alone. Tried seeded Branch-Train-MiX first then dropped it. Adds compute for no clear win when the router will find its own attractors anyway.
Big lesson today came from limit testing on A100 80GB. Surprise, every planned phase ran out of memory even on 80GB. Root cause: at vocab 262144 (Gemma 3 standard), the output logits dominate during forward and backward. Fix: Liger Kernel's fused cross-entropy. It streams the loss computation instead of materialising the full B by T by vocab tensor. Without it the build would not run.
Scamp proved the pipeline runs end-to-end on real hardware. The 412M run starts tomorrow. If routing balances naturally and math finally crystallises, ships as Crowfeather-412M-3E with GGUF in F16, Q8, Q5, and Q4.
So... the training may have produced a poet if I had done it better. But I didn't, so instead... we get a malformed robot named Scamp... This is progress.
-Shane
P.S Join discord for discussion: https://discord.gg/8ZscHNmJYE and
I post my finished stuff here: CompactAI-O
Scamp ships, hits the wall. New plan...
Scamp came back from training today... Didn't go so well, I'm still unsure...
Fast benchmark, temperature 0.7, top_p 0.9:
- "Capital of France is" produced "covered by the Crown" (grammatical, factually wrong)
- "23 + 19 = ?" produced "23. Answer: 23. Answer: 23..." (loops, math broken)
- "def fibonacci(n):" produced a list of letters
It speaks English. It can't reason. At 8K vocab and 50M params, it was never going to.
Next build: 412M MoE-3E. Three experts (math, language, code), top-1 routing, random init, let specialization emerge from gradient signal alone. Tried seeded Branch-Train-MiX first then dropped it. Adds compute for no clear win when the router will find its own attractors anyway.
Big lesson today came from limit testing on A100 80GB. Surprise, every planned phase ran out of memory even on 80GB. Root cause: at vocab 262144 (Gemma 3 standard), the output logits dominate during forward and backward. Fix: Liger Kernel's fused cross-entropy. It streams the loss computation instead of materialising the full B by T by vocab tensor. Without it the build would not run.
Scamp proved the pipeline runs end-to-end on real hardware. The 412M run starts tomorrow. If routing balances naturally and math finally crystallises, ships as Crowfeather-412M-3E with GGUF in F16, Q8, Q5, and Q4.
So... the training may have produced a poet if I had done it better. But I didn't, so instead... we get a malformed robot named Scamp... This is progress.
-Shane
P.S Join discord for discussion: https://discord.gg/8ZscHNmJYE and
I post my finished stuff here: CompactAI-O
- 2 replies
reacted to Crownelius's post with 🔥 about 2 months ago
Post
3685
[DAY TWO] PROJECT CROWFEATHER - 5/1/2026
Que sera, what will he be?
Step 47,500 of 100,000. Loss hovering around 2.76 on 6.2B tokens. Throughput steady at 87k per second on the A100. Not a GH200, but she gets it done.
Still haven't named him. Scamp has a rascally charm. Quentin sounds like he'd wear a bow tie and think hard before speaking. Taking votes.
Phase two is what's keeping me up. Datasets everywhere and I can't pick. I'm fusing Google and DeepSeek's ideas: Gemma 4's alternating sliding and global attention, DeepSeek V4's Muon optimizer and WSD scheduler, Gemma 2's logit soft cap, and PaLM's z-loss. Sounds like peanut butter on a hamburger, but the loss curve says it works.
Tribe_v2 has real potential but needs more scaffolding than a barn raising before I throw it in. One thing's certain though. This model's gonna be a thinker. Not a Wikipedia parrot. Something that chews before it answers.
Finally got a use for my less popular datasets too. Some Opus-4.5-Writing-Style for polish. A few rows of Human-Archtypes-25k to see what personality bubbles up. Could be a poet, could be a grump. Either beats a flimsy fine-tune.
The bank's after my credit card. Until then, full steam.
Next model gets graphs. I swear.
-Shane
Que sera, what will he be?
Step 47,500 of 100,000. Loss hovering around 2.76 on 6.2B tokens. Throughput steady at 87k per second on the A100. Not a GH200, but she gets it done.
Still haven't named him. Scamp has a rascally charm. Quentin sounds like he'd wear a bow tie and think hard before speaking. Taking votes.
Phase two is what's keeping me up. Datasets everywhere and I can't pick. I'm fusing Google and DeepSeek's ideas: Gemma 4's alternating sliding and global attention, DeepSeek V4's Muon optimizer and WSD scheduler, Gemma 2's logit soft cap, and PaLM's z-loss. Sounds like peanut butter on a hamburger, but the loss curve says it works.
Tribe_v2 has real potential but needs more scaffolding than a barn raising before I throw it in. One thing's certain though. This model's gonna be a thinker. Not a Wikipedia parrot. Something that chews before it answers.
Finally got a use for my less popular datasets too. Some Opus-4.5-Writing-Style for polish. A few rows of Human-Archtypes-25k to see what personality bubbles up. Could be a poet, could be a grump. Either beats a flimsy fine-tune.
The bank's after my credit card. Until then, full steam.
Next model gets graphs. I swear.
-Shane
- 3 replies
reacted to Crownelius's post with 🧠😎 about 2 months ago
Post
3835
[DAY ONE] PROJECT CROWFEATHER 4/30/2026
...The day I forgot to attach wandb.ai
Just dropped Crowfeather-50m, the first checkpoint in a series, and yeah, no graphs.
Crowfeather/Crowfeather-50m
54.5M params. Pretrain only. 17,500 steps banked on FineWeb-edu before Thunder credits ran dry. About 2.3B tokens, no SFT yet.
Architecture: Gemma-4 alternating sliding/global attention (1024 window, last layer always global) plus DeepSeek-V4 Muon optimizer plus WSD scheduler plus Gemma-2 logit soft-cap plus PaLM z-loss. Recipe in the model card.
What it can do: writes grammatical English. Knows that France has Rhine-adjacent monasteries (it picked Rouen instead of Paris but the vocabulary is in there). Tells stories about Mr. Fabien.
What it can't do yet: facts, code, math. Base LM, no SFT, no instruction tuning.
The series:
Every additional training run becomes another model card here
Every model card gets a matching post on this profile
Continuation goes to Colab next, picking up from step 17500 out of 100k
Limited to one post a day on Hugging Face, so updates will trickle out at that pace. Follow [@Crownelius](@Crownelius ) and [@Crowfeather](
Crowfeather ) if you want to watch this thing learn in public. Next drop will either come with the finished pre-train or whatever step I land on before the bank takes my credit card away.
Graphs will be available on my NEXT model lol
-Shane
...The day I forgot to attach wandb.ai
Just dropped Crowfeather-50m, the first checkpoint in a series, and yeah, no graphs.
Crowfeather/Crowfeather-50m
54.5M params. Pretrain only. 17,500 steps banked on FineWeb-edu before Thunder credits ran dry. About 2.3B tokens, no SFT yet.
Architecture: Gemma-4 alternating sliding/global attention (1024 window, last layer always global) plus DeepSeek-V4 Muon optimizer plus WSD scheduler plus Gemma-2 logit soft-cap plus PaLM z-loss. Recipe in the model card.
What it can do: writes grammatical English. Knows that France has Rhine-adjacent monasteries (it picked Rouen instead of Paris but the vocabulary is in there). Tells stories about Mr. Fabien.
What it can't do yet: facts, code, math. Base LM, no SFT, no instruction tuning.
The series:
Every additional training run becomes another model card here
Every model card gets a matching post on this profile
Continuation goes to Colab next, picking up from step 17500 out of 100k
Limited to one post a day on Hugging Face, so updates will trickle out at that pace. Follow [@Crownelius](@Crownelius ) and [@Crowfeather](
Graphs will be available on my NEXT model lol
-Shane
- 3 replies