Kudos on Atom 2.7M
I just wanted to say I'm a big fan of what you've developed with Atom 2.7M. I always saw Arithmark-2 as one of those benchmarks that could only be beaten with scale and curated synthetic data, and you went and showed me how wrong I was with an inspired architecture. I'm really looking forward to seeing what else you come up with.
Thank you, that means a lot. I’m really excited about where this is headed and can’t wait to share more future projects soon.
I did try an Atom model at around 500k parameters, but it struggled to handle language modeling and arithmetic simultaneously. An arithmetic-only Transformer at roughly the same parameter scale was successful, though, and I’ll link the article below:
https://huggingface.co/blog/ucr-max/atom2-7m-arithmetic-representation
My focus has now shifted toward more commonsense-oriented pretraining objectives, especially alternative tokenization approaches and datasets with higher educational density.