Upload README.md with huggingface_hub

README.md

@@ -0,0 +1,138 @@
+---
+license: agpl-3.0
+library_name: pytorch
+tags:
+- tiny-lm
+- goldfish
+- transformer
+- rope
+- swiglu
+pipeline_tag: text-generation
+base_model: []
+---
+
+# GlubLM (36M)
+
+> *the language model that already forgot this sentence*
+
+**GlubLM** is a 36-million-parameter transformer that plays the character of a goldfish with a 10-second memory. Inspired by [GuppyLM](https://github.com/arman-bd/guppylm) by Arman BD and Ted Lasso's meditation on the goldfish as "the happiest animal on earth", GlubLM has a hard 96-token context window - it *physically* cannot remember what was just said.
+
+Try it live: [browser demo](https://den-sec.github.io/glublm/) | [pixel-art desk pet](https://den-sec.github.io/glublm/desk-pet/)
+
+## Architecture
+
+- **Parameters**: 36,055,680 (36.1M)
+- **Layers**: 8 decoder-only transformer blocks
+- **Hidden dim**: 640
+- **Attention heads**: 10 (head dim 64)
+- **FFN dim**: 1280 (SwiGLU, effective intermediate 2560)
+- **Normalization**: RMSNorm
+- **Position encoding**: Rotary (RoPE)
+- **Vocabulary**: 5,120 Byte-Level BPE
+- **Max context**: 96 tokens (hard cap, the "10-second memory")
+- **Weight-tied LM head**
+- **No bias terms**
+
+## Intended use
+
+This model is a toy. It exists to:
+1. Explore the design tension between "small + simple" (GuppyLM's thesis) and "small + modern" (GlubLM's hypothesis)
+2. Demonstrate an LLM-generated dataset pipeline using a multi-agent Claude team
+3. Be a fun browser demo and a pixel-art desk pet companion
+
+**Do not use GlubLM for anything serious.** It literally forgets within a sentence.
+
+## Training data
+
+Trained on [`DenSec02/glublm-60k-ted`](https://huggingface.co/datasets/DenSec02/glublm-60k-ted), a 60,549-sample dataset of single-turn goldfish conversations generated by a team of four coordinated Claude agents (generator, critic, diversifier, persona-guardian). Composition: v4 balanced mix (20K poetic + 15K supplement + 5K conversational + 15K forgetful) augmented with v5.1 empathic/introspective hotfix (1K samples) + v5.2 multi-anchor self-awareness recovery (500 samples).
+
+**Explicit exclusions**: no references to football, soccer, coaches, teams, or any Ted Lasso show characters.
+
+## Training
+
+- **Hardware**: NVIDIA RTX 3060 12GB (local)
+- **Framework**: PyTorch 2.x, BF16 mixed precision
+- **Optimizer**: AdamW (b1=0.9, b2=0.95), weight decay 0.1
+- **LR schedule**: cosine with 5% warmup, peak 3e-4
+- **Batch size**: 64
+- **Epochs**: 15
+- **Dropout**: 0.1 (residual), 0.0 (attention)
+- **Gradient clipping**: 1.0
+- **Final loss**: 1.1442
+- **Wall time**: ~15 minutes
+
+## Evaluation (v2 cross-model judge)
+
+Dual-judge evaluation using Claude Sonnet 4.6 and Opus 4.7 on a 30-prompt rubric across 4 axes (integer 1-5 scale). Each axis aggregates 30 prompts x 3 seeds x 2 passes = 180 scoring rows per judge.
+
+### Per-axis score (mean)
+
+| Axis | Sonnet 4.6 | Opus 4.7 |
+|---|---:|---:|
+| Conversational Quality | 4.01 | 4.15 |
+| Goldfish Identity | 3.89 | 3.67 |
+| Forgetful Trait | 3.80 | 3.81 |
+| Length Appropriateness | 4.77 | 4.57 |
+
+### Cross-judge agreement (Cohen's quadratic-weighted kappa)
+
+| Axis | Kappa | Interpretation |
+|---|---:|---|
+| Conversational Quality | 0.77 | substantial |
+| Goldfish Identity | 0.83 | almost perfect |
+| Forgetful Trait | 0.86 | almost perfect |
+| Length Appropriateness | 0.59 | moderate |
+
+**Interpretation**: Sonnet and Opus agree almost perfectly on 3/4 axes, validating that the rubric is interpretable consistently across LLM judges. Opus tends to be systematically ~0.2 stricter than Sonnet on the Identity axis (stricter rubric application, not judge bias).
+
+Full methodology + 108-row long-format scores: [`eval/report_crossmodel.md`](https://github.com/Den-Sec/glublm/blob/master/eval/report_crossmodel.md).
+
+## Limitations & biases
+
+- **Hard context limit**: 96 tokens. Inputs longer than a few short sentences will be truncated.
+- **Goldfish worldview**: the model genuinely does not understand human abstractions outside the bowl.
+- **Dataset bias**: the dataset was generated by Claude (Anthropic), so it inherits Claude's language patterns filtered through the goldfish persona.
+- **Single-turn only**: multi-turn memory is a non-goal.
+- **English only**.
+- **Stochastic and occasionally incoherent**: 36M params on 60K samples is small. Do not expect reliability.
+
+## How to use
+
+```python
+from glublm.config import ModelConfig
+from glublm.model import GlubLM
+from glublm.tokenizer import GlubTokenizer
+from glublm.inference import generate
+from huggingface_hub import hf_hub_download
+from safetensors.torch import load_model
+
+tok_path = hf_hub_download("DenSec02/glublm-36m", "tokenizer.json")
+weights_path = hf_hub_download("DenSec02/glublm-36m", "model.safetensors")
+
+tok = GlubTokenizer.from_file(tok_path)
+cfg = ModelConfig(vocab_size=tok.vocab_size)
+model = GlubLM(cfg)
+load_model(model, weights_path)
+
+print(generate(model=model, tokenizer=tok, prompt="hello", max_new_tokens=24))
+```
+
+Or try it in-browser with zero setup:
+- [Chat demo](https://den-sec.github.io/glublm/) (simple web UI)
+- [Desk pet companion](https://den-sec.github.io/glublm/desk-pet/) (pixel-art PWA)
+- [Colab notebook](https://colab.research.google.com/github/Den-Sec/glublm/blob/master/notebooks/train_colab.ipynb) (train your own goldfish)
+
+## License
+
+AGPL-3.0 - see [LICENSE](https://github.com/Den-Sec/glublm/blob/master/LICENSE).
+
+## Citation
+
+```bibtex
+@software{glublm_2026,
+  author       = {Sepede, Dennis},
+  title        = {GlubLM: a 36M goldfish language model with a 10-second memory},
+  year         = {2026},
+  url          = {https://github.com/Den-Sec/glublm}
+}
+```