ace-1
/

mgpt2-tokenizer

Model card Files Files and versions

YAML Metadata Warning: empty or missing yaml metadata in repo card (https://huggingface.co/docs/hub/model-cards#model-card-metadata)

ace-1/mgpt2-tokenizer

Custom mgpt2 tokenizer (pure-Python) exported for Hugging Face trust_remote_code=True.

Usage

from transformers import AutoTokenizer

tok = AutoTokenizer.from_pretrained('ace-1/mgpt2-tokenizer', trust_remote_code=True)
print(tok.encode('hello world'))

Contents

Trained tokenizer artifact: mgpt2.model (native .model format)
Python implementation under tokenizer/ (loaded via trust_remote_code)

Downloads last month: -; Downloads are not tracked for this model. How to track

Inference Providers NEW

This model isn't deployed by any Inference Provider. 🙋 Ask for provider support