Shaping capabilities with token-level data filtering
Paper
•
2601.21571
•
Published
•
25
ZeroGPU Explorers
community
AI & ML interests
None defined yet.
Recent Activity
View all activity
julien-c
submitted
a
paper
to Daily Papers
11 days ago
innovation64
authored
a
paper
12 days ago
innovation64
submitted
a
paper
to Daily Papers
12 days ago
Post
3082
Reverse Engineering a $500M Mystery: From HashHop to Memory-Augmented Language Models
I wrote a deep dive into how Magic AI's 100M token context window might work, starting from their HashHop benchmark and building up to MALM - a Memory-Augmented Language Model.
Key insight: treating each key as a single token enables perfect retrieval at unlimited context lengths.
The article covers:
- How HashHop works and why its perfect accuracy is suspicious
- Building a tokenized solver that achieves 100% accuracy
- Scaling to MALM for real code search tasks
- Why this approach could handle 100M+ tokens
Read the full article: https://huggingface.co/blog/codelion/reverse-engineering-magic-hashhop
Try the model: codelion/malm-165m
Code: https://github.com/codelion/hash-hop
I wrote a deep dive into how Magic AI's 100M token context window might work, starting from their HashHop benchmark and building up to MALM - a Memory-Augmented Language Model.
Key insight: treating each key as a single token enables perfect retrieval at unlimited context lengths.
The article covers:
- How HashHop works and why its perfect accuracy is suspicious
- Building a tokenized solver that achieves 100% accuracy
- Scaling to MALM for real code search tasks
- Why this approach could handle 100M+ tokens
Read the full article: https://huggingface.co/blog/codelion/reverse-engineering-magic-hashhop
Try the model: codelion/malm-165m
Code: https://github.com/codelion/hash-hop
- 1 reply
LXT
submitted
a
paper
to Daily Papers
19 days ago
BK-Lee
authored
a
paper
about 1 month ago
BK-Lee
submitted
a
paper
to Daily Papers
about 1 month ago
Post
6085
Introducing Dhara-70M: A diffusion language model that achieves 3.8x higher throughput than autoregressive models!
Key findings from our research on optimal architectures for small language models:
→ Depth beats width: 32 layers outperforms 12 layers at the same parameter count
→ Best-in-class factuality: 47.5% on TruthfulQA
→ 10x training efficiency using WSD (Warmup-Stable-Decay) conversion
→ Canon layers add only 0.13% parameters but improve reasoning
We trained on 1B tokens using the optimal 50-30-20 dataset mix (PDFs + filtered web + educational content), then converted to diffusion with just 100M additional tokens.
Blog: https://huggingface.co/blog/codelion/optimal-model-architecture
Model: codelion/dhara-70m
Key findings from our research on optimal architectures for small language models:
→ Depth beats width: 32 layers outperforms 12 layers at the same parameter count
→ Best-in-class factuality: 47.5% on TruthfulQA
→ 10x training efficiency using WSD (Warmup-Stable-Decay) conversion
→ Canon layers add only 0.13% parameters but improve reasoning
We trained on 1B tokens using the optimal 50-30-20 dataset mix (PDFs + filtered web + educational content), then converted to diffusion with just 100M additional tokens.
Blog: https://huggingface.co/blog/codelion/optimal-model-architecture
Model: codelion/dhara-70m
- 1 reply
ShoufaChen
authored
a
paper
about 2 months ago
Post
2405
Introducing PTS Visualizer - an interactive tool for exploring how language models reason!
Visualize pivotal tokens, thought anchors, and reasoning circuits. See which tokens and sentences significantly impact success probability, explore embedding clusters, and trace reasoning step-by-step.
Try it: codelion/pts-visualizer
Explore PTS datasets:
- Qwen3-0.6B: codelion/Qwen3-0.6B-pts
- DeepSeek-R1: codelion/DeepSeek-R1-Distill-Qwen-1.5B-pts
Or upload your own JSONL files!
GitHub: https://github.com/codelion/pts
Visualize pivotal tokens, thought anchors, and reasoning circuits. See which tokens and sentences significantly impact success probability, explore embedding clusters, and trace reasoning step-by-step.
Try it: codelion/pts-visualizer
Explore PTS datasets:
- Qwen3-0.6B: codelion/Qwen3-0.6B-pts
- DeepSeek-R1: codelion/DeepSeek-R1-Distill-Qwen-1.5B-pts
Or upload your own JSONL files!
GitHub: https://github.com/codelion/pts
LXT
authored
10
papers
about 2 months ago
DenseWorld-1M: Towards Detailed Dense Grounded Caption in the Real World
Paper
•
2506.24102
•
Published
One Flight Over the Gap: A Survey from Perspective to Panoramic Vision
Paper
•
2509.04444
•
Published
VimoRAG: Video-based Retrieval-augmented 3D Motion Generation for Motion Language Models
Paper
•
2508.12081
•
Published
DiT360: High-Fidelity Panoramic Image Generation via Hybrid Training
Paper
•
2510.11712
•
Published
•
31
Grasp Any Region: Towards Precise, Contextual Pixel Understanding for Multimodal LLMs
Paper
•
2510.18876
•
Published
•
37
Open-o3 Video: Grounded Video Reasoning with Explicit Spatio-Temporal Evidence
Paper
•
2510.20579
•
Published
•
56
From Masks to Worlds: A Hitchhiker's Guide to World Models
Paper
•
2510.20668
•
Published
•
8
PairUni: Pairwise Training for Unified Multimodal Language Models
Paper
•
2510.25682
•
Published
•
14
Are Video Models Ready as Zero-Shot Reasoners? An Empirical Study with the MME-CoF Benchmark
Paper
•
2510.26802
•
Published
•
34
Visual Spatial Tuning
Paper
•
2511.05491
•
Published
•
52