AbstractPhil/alephllm-chat-history
Viewer • Updated • 1
AbstractPhila PRO
AbstractPhil
AI & ML interests
datasets, research papers, experimentation, vision, classification, text encoders, tokenization, llms, diffusion, distillation, and more.
Recent Activity
posted an update about 5 hours ago
Say hello to the AlephLLM: Mini-Beatrix - in her huggingface space!
She is currently stepped at 8000 steps in the first couple datasets, so she's not very smart yet.
https://huggingface.co/spaces/AbstractPhil/alephllm-chat
Be warned, whatever you say will be recorded in a public cache.
The AlephLLM prototype is currently in full training with SDPA attention.
https://huggingface.co/AbstractPhil/alephllm-mini-beatrix-training
https://github.com/AbstractEyes/alephllm Here's the model code and training code for the prototype.
As the training progresses, the AlephLLM will become more coherent and communicative, the tensorboard will consist of a large series of useful and useless analysis, and each checkpoint recorded at around 2000 steps unless the train crashes or the system faults.
It will take about 9 hours for the first few datasets to converge, then I'll train a chat AMOE expert cluster to see if she wants to speak yet. Until then, she's learning.
Yes I know it's early, but there isn't much more I could think of to analyze the AlephLM directly currently. The only way train the AlephLLM, is to train the full AlephLLM prototype. The bigger training has to run, otherwise the analysis won't matter. As it progresses, the analysis and huge amount of tensorboard statistics will flood out. Everything is transparent through the process from start to finish, everything recorded. updated a model about 5 hours ago
AbstractPhil/alephllm-mini-beatrix-training repliedto their post about 7 hours ago
The upcoming AlephLM LLM prototype "Mini-Beatrix" is based on protocols, rules, and laws established through the process of training AlephLM systems. This will be a first attempt at a smaller full pretrain/finetune of the AlephLLM on raw data, and this will require over a billion unigram tokens.
Mini-Beatrix will inherit an appropriately adapted AlephLM MOE structure containing a multitude of trained experts, a gating system, a long context RoPE system, MHA attention, and a series of hypothesis to answer upon Mini-Beatrix's pretrain and finetune completion.
While focusing on resolving corruptions and invalidity possibly present in the splat attention, the solutions raised SDPA attention protocol token recall ceiling from 0.91 to 0.993. With that the splat attention raised from 0.81 to 0.89~ splat being around 3x the speed is still imperfect.
So far so good. The corruptions have resolved multiple core component overlapping problems causing the AlephLM's inability to handle the trigram system, the structure of the SVAE having faulty trigram structures, and additionally a multitude of other systems in the lineup that were inheriting the corruptions from the core experiment sets.
These corruptions resolved show that the accuracy of standard multiheaded attention will provide the necessary token recall for full LM capacity, and with that If and WHEN I solve the Rorschach Splat attention will be the faster alternative at >=r1 0.99%, only then. The splat attention's considerably larger head count still contains unresolved inconsistencies.
That being said the SDPA MHA attention will be present for the first attempted mini-llm train, which will be named "Mini-Beatrix" with the appropriate sizing associated with this.
The only thing that will change Mini-Beatrix's trajectory will be if Splat attention is perfected between today and next week, which will likely take longer unless I run into a core corruption that has been overlooked through hundreds of analysis.Organizations
