Any chance of shipping a tokenizer.json (fast tokenizer) with this checkpoint?
The repo currently ships the SentencePiece tokenizer.model plus the custom tokenization_internlm3.py, but no tokenizer.json. That works fine from Python with trust_remote_code=True, but it locks out the non-Python ecosystem: swift-transformers (the tokenizer layer used by on-device Swift apps on Apple silicon) loads tokenizers exclusively from tokenizer.json and has no SentencePiece fallback, so InternLM3 currently can't be tokenized there at all. The MLX conversions on the Hub (mlx-community/internlm3-8b-instruct-4bit etc.) inherit the same gap, since the conversion just copies the tokenizer files.
Would you consider adding a tokenizer.json to the repo? For most SentencePiece models it's a one-time AutoTokenizer.from_pretrained(...).save_pretrained(...) with the fast tokenizer enabled on your side — and since it would come from you rather than a third party, downstream users get a tokenizer that's guaranteed to match the slow one. Happy to help verify it against the slow tokenizer's output if useful.
Thanks for your issue.
Unfortunately, AutoTokenizer.from_pretrained("...", trust_remote_code=True, use_fast=True).save_pretrained(...) won't produce a tokenizer.json in this case. To the best of my knowledge, SentencePiece applies a longest-match strategy, while transformers-tokenizers relies on rule-based merging, which can lead to unavoidable discrepancies between the slow and fast tokenizers.
We've tried converting the SentencePiece model to a fast tokenizer, but the result does not exactly match the slow tokenizer. Therefore, we feel it is not appropriate to provide an official but flawed alternative.
For reference, the conversion script we experimented with may be helpful: tokenizers/bindings/python/scripts/convert.py
We will continue looking into whether there is a better solution. If you have any ideas or suggestions, we'd be very happy to discuss them.