Article
A ring of teachers, a 1.7B brain, and a harness that couldn't lie
PeetPedro
• Dipankar Sarkar PRO
dipankarsarkar
AI & ML interests
Building the AI-native stack. Agents as infrastructure, safety as architecture, performance as plumbing. I publish the receipts: papers, datasets, demos.
Recent Activity
repliedto nightmedia's post 5 minutes ago
Qwen3.8-27B metrics
It's hard to track all model cards where I post these, so I figured people would get more value out of seeing these in the open.
```brainwaves
quant arc arc/e boolq hswag obkqa piqa wino
mxfp8 0.591,0.782,0.896,0.746,0.448,0.801,0.711
q8-hi 0.602,0.779,0.896,0.747,0.446,0.793,0.703
q6-hi 0.602,0.775,0.895,0.748,0.448,0.795,0.710
q4-hi 0.604,0.780,0.898,0.744,0.454,0.795,0.708
mxfp4 0.581,0.771,0.889,0.738,0.442,0.798,0.713
Quant Perplexity Peak Memory Tokens/sec
mxfp8 6.090 ± 0.054 34.74 GB 138
mxfp4 5.952 ± 0.051 21.30 GB 148
```
The performance is as measured on a M4 MBP 128GB, speed may vary depending on your platform.
These are all instruct metrics, generated by including this line in the jinja template:
```
{%- set enable_thinking = false %}
```
Then run the test suite to generate the metrics:
```
mlx_lm.evaluate --model MODEL --tasks winogrande boolq arc_challenge arc_easy hellaswag openbookqa piqa
```
This will generate the file:
```
eval_MODEL_0.4.9_winogrande_boolq_arc_challenge_arc_easy_hellaswag_openbookqa_piqa
```
This is a JSON containing all gathered metrics; for example the q4-hi:
```
"arc_challenge": {
"alias": "arc_challenge",
"acc,none": 0.5819112627986348,
"acc_stderr,none": 0.014413988396996116,
"acc_norm,none": 0.6040955631399317,
"acc_norm_stderr,none": 0.01429122839353657
},
```
I use the value of acc_norm for metrics, rounded to 3 decimals.
As I get more quants tested, I will add them here.
A complete test run for a single quant takes 7-9 hours depending on quant size, 10-12 hours for BF16 depending on perplexity: this is why you see on my model cards that I usually post the first three, that only take 2-3 hours :)
-G upvoted a paper 5 minutes ago
Temporal Multi-Signal Fusion for Token-Level Hallucination Detection repliedto davidmezzetti's post 14 minutes ago
Exciting addition coming with the next txtai release: LEMUR for ColBERT-style Late-Interaction Retrieval! 🎉
Contributor @Morgan-coded introduced LEMUR to txtai, making it, as far as we know, the first framework to incorporate LEMUR for late-interaction retrieval using standard, fixed-vector indexes.
Key benefits:
🚀 Significant boost: 49–62% higher NDCG@10 than 2,048-dimensional MUVERA
💾 5x less storage: 2,048 dimensions vs. MUVERA’s default 10,240
📐 Better geometry: Optional batch mean centering addresses anisotropy in token embeddings
A promising step toward making ColBERT-style retrieval more practical with conventional vector search.
Read the full breakdown: https://huggingface.co/blog/NeuML/txtai-lemurOrganizations
published an article 4 days ago
published an article 5 days ago
Article
CLASSROOM-SOTA training spec — ring-of-teachers distillation for the quantal brain
PeetPedro
• published an article 7 days ago
Article
quantal-ternary: a 0.5B model that learned honesty — from 11.34 to 0.5597
PeetPedro
• published an article about 2 months ago
Article
The cheap oracle is lying to you: from GPU kernels to agent rewards
dipankarsarkar
• • 2