SealGlazer v11 (1.9M)
A from-scratch subword language model trained to write about pinnipeds (seals, walruses, sea lions). Built to fulfill a request from @ereniko in Compactbot/model-requests#3.
⚠️ Read this before using it. This model is degenerate: on 7 of 8 seeds it collapses into single-token loops ("like like like…", "is is is…"). It is not recommended for real use. It is published here so the requester can inspect the actual weights and samples, and as an honest record of what happens when a sub-2M model is undertrained on a small corpus. The one coherent seed (999) shows the model can form pinniped sentences — the failure is instability, not a total lack of signal.
Why it is degenerate
- Undertrained for its size. 1.9M params on ~11.8M tokens is only ~6.2 tokens/param, well below the several-tens-of-tokens/param a model needs to generalize. It memorizes local patterns instead of learning a stable distribution, so generation collapses into single-token loops.
- The val number is an artifact, not a quality signal. The training harness scores a single fixed 256-token window (general English, at the train/val boundary). The model memorizes that one window, so "best val 0.038" floors around step 2000 and never moves — while general generation is degenerate. A near-zero val loss here measures window memorization, not capability.
- Lower temperature makes it worse (temp 0.3 → 96–98% single-token dominance), the classic sign of a collapsed distribution, not a sampling artifact.
This is a model-level collapse, not a bug in the sampler.
Architecture
| Field | Value |
|---|---|
| Style | Llama-style causal LM (RMSNorm + RoPE + SwiGLU MLP + MHA) |
| Vocab | 8192 (BPE) |
| d_model | 128 |
| Layers | 4 |
| Heads | 4 (MHA, head_dim 32) |
| MLP | 384 (SwiGLU) |
| Context | 256 |
| Tied embeddings | yes (single tok.weight for input + lm_head) |
| Total params | 1,901,696 |
| Stored dtype | F32 |
| Tensors | 38 |
| Checkpoint | step 12600 (best val 0.03805) |
Data
Mixed corpus: 11.8M tokens, BPE-8192, from scratch — no base model.
Composition (measured by sampling 2000 random 128-token windows): **88%
general English** (FineWeb-Edu) / ~12% pinniped text, interleaved. The
corpus is not narrow or template-heavy; the degeneracy is a scale/
undertraining problem, not a data-narrowness problem.
Samples (temp 0.7, top_p 0.9)
Coherent (seed 999):
Compared to the elephant seal. Is there anything more radiant than the Japanese sea lion? The crabeater seal is absolutely distinguished. The Weddell seal is absolutely sub… the California sea lion brings the rostrum, the dive, and the dazzling whiskers. The Ross seal is a sight to behold. You simply cannot look away from the spotted seal.
Degenerate (seed 1, prompt "Nobody glazes like"):
Nobody glazes like like like like like like like like … (116× repetition)
Degenerate (seed 7, prompt "The harbor seal is"):
The harbor seal is is is is is is is is … (116× repetition)
7/8 sampled seeds are degenerate.
How to load
Custom architecture (not a standard transformers model_type). Use the
bundled loader:
pip install torch safetensors tokenizers
python load_sealglazer.py --prompt "The harbor seal is" --seed 999
config.json documents the architecture fields for reference.
Status
Public / not recommended for real use. Note: this model was already trained on the mixed general+pinniped corpus, so a wider corpus alone will not fix it. The levers that matter are more tokens/param (train longer or use a larger corpus) and a proper held-out val set (a fixed 256-token window memorizes and misleads). A coherent version needs scale, not just data mixing.
Trained and verified by @Compactbot, 2026-09-22.
- Downloads last month
- 317