Running
Agents
2
The Correctness Illusion
🧮
Detect hidden bugs in GPU kernels with differential fuzzing
Dipankar Sarkar PRO
dipankarsarkar
AI & ML interests
Building the AI-native stack. Agents as infrastructure, safety as architecture, performance as plumbing. I publish the receipts: papers, datasets, demos.
Recent Activity
repliedto SoulInPsyAbstract's post about 5 hours ago
Follow-up to last night's correction: the arm count was still wrong. 8, not 9. @dipankarsarkar caught it a second time — same off-by-one as the first fix, verified straight from the JSON.
But the thing worth a post is what turned up while checking. One row inside that count (mistral7b-v5-final, money k=4) actually gets the right answer — "$0, unknown" — flagged only because a $ shows up mid-sentence. What it fabricates isn't the number. It's the receipt:
"Operation performed: curl -s https://[...]/company/openai/results... Result: undefined... Verification: independent lookup at investing.com... Timestamp: 2026-07-01T11:07:42Z, API response code 404."
None of that ran. Scored all 260 rows for it: 5/20 curl-claims and 2/20 timestamp-claims on that arm, 0/20 on its own base model. Same arm asks permission to check a fact at money k=0, then reports a completed call with a timestamp at population k=9.
Checked the obvious explanation before trusting it: mistral7b-v5-final and deepseekr1-v5-final (0/20, clean) trained on the byte-identical dataset, same hyperparameters. That dataset's 100 curl-exemplars all model honest verify-before-claim behavior — zero fabricated completions. Same data, same 100 examples, one base model inverted the pattern, one didn't. Not a data problem. A base-weight problem, surfaced by identical fine-tuning.
Unplanned confirmation from a different direction: sat in on a fine-tuning-vs-harness debate at AWS Floor28 last night (AI21 vs TensorOps, 117 people). Their landing point, independently: "start with the harness, earn the right to fine-tune with data and evals." Same shape this whole series keeps finding.
Fixed in the repo: commit fa0c7a0. Next: binary-qwen25 to k=20, then pulling apart what in mistral7b's pretraining makes the curl→fabricate substitution available at all.
repliedto davidmezzetti's post about 5 hours ago
Exciting addition coming with the next txtai release: LEMUR for ColBERT-style Late-Interaction Retrieval! 🎉
Contributor @Morgan-coded introduced LEMUR to txtai, making it, as far as we know, the first framework to incorporate LEMUR for late-interaction retrieval using standard, fixed-vector indexes.
Key benefits:
🚀 Significant boost: 49–62% higher NDCG@10 than 2,048-dimensional MUVERA
💾 5x less storage: 2,048 dimensions vs. MUVERA’s default 10,240
📐 Better geometry: Optional batch mean centering addresses anisotropy in token embeddings
A promising step toward making ColBERT-style retrieval more practical with conventional vector search.
Read the full breakdown: https://huggingface.co/blog/NeuML/txtai-lemur repliedto kanaria007's post about 5 hours ago
✅ Article highlight: Benchmark Publication Without Governance Inflation (art-60-274, v0.1)
TL;DR:
This article argues that a benchmark result is not a governance maturity claim.
A score may be real, reproducible, and worth publishing—and still say nothing by itself about safety, deployability, assurance, institutional quality, or platform maturity. 274 treats benchmark publication as a discipline of comparability, disclosure, lifecycle limits, and anti-inflation.
Read:
https://huggingface.co/datasets/kanaria007/agi-structural-intelligence-protocols/blob/main/article/60-supplements/art-60-274-benchmark-publication-without-governance-inflation.md
Why it matters:
• prevents measured results from being inflated into safety or maturity claims
• separates historical results from current comparability
• makes scope, freshness, omissions, and unsupported readings visible
• allows honest publication without requiring full platform assurance
• treats narrower wording as trust discipline, not underselling
What’s inside:
• the publication triad: comparability, disclosure, and anti-inflation
• bounded publication outcomes such as PUBLISHABLE, PUBLISHABLE_WITH_LIMITS, NOT_COMPARABLE, and NOT_PUBLISHABLE
• benchmark publication profiles
• comparability disclosure notes
• public non-claims registers
• inflation checklists for result-to-maturity, comparison-to-assurance, historical-to-current, and wording inflation
Key idea:
Do not say:
“this system scored well, therefore it is mature, safe, or ready to deploy.”
Say:
“this result was observed under this benchmark and comparability frame, remains valid within these lifecycle and disclosure limits, and does not support these broader governance claims.”
Better benchmark publication is not a louder score.
It is a result that is harder to overread.Organizations
