Is there a way to package this? #21

Activity

1. PythonNut commented on Jan 17, 2026

   

   PythonNut

   on Jan 17, 2026

   Last edited by PythonNut

   Owner

   More actions

   This repo is primarily intended to be used for training. Our expectation is that people will generally want to modify/tweak the training scripts before running them, so I don't think packaging them makes sense.

   For inference, you can use the original tokenizers library. (You don't need this repo.)

   Additionally, the tokenizers SuperBPE fork can be installed via its git address via pip or any other python package manager.