Skip to content

Add missing return type hints to ByteLevelBPETokenizer.from_file, train, train_from_iterator - #2326

Closed
RudrenduPaul wants to merge 1 commit into
huggingface:mainfrom
RudrenduPaul:fix/byte-level-bpe-return-type-hints
Closed

RudrenduPaul wants to merge 1 commit into
huggingface:mainfrom
RudrenduPaul:fix/byte-level-bpe-return-type-hints

Conversation

@RudrenduPaul

Copy link
Copy Markdown

What this PR does

Adds missing return type hints to three methods in
bindings/python/py_src/tokenizers/implementations/byte_level_bpe.py:

  • from_file(vocab_filename: str, merges_filename: str, **kwargs) -> "ByteLevelBPETokenizer"
  • train(...) -> None
  • train_from_iterator(...) -> None

This follows the same typing-consistency pattern as #2211, which added
the equivalent missing return hints to BaseTokenizer.save,
save_model, and to_str. ByteLevelBPETokenizer (a BaseTokenizer
subclass) has the identical gap: every argument was already annotated,
but the return type was never added to these three methods.

Why these types are correct

  • from_file is a @staticmethod that returns
    ByteLevelBPETokenizer(vocab, merges, **kwargs), i.e. a new instance
    of the class — hence -> "ByteLevelBPETokenizer", using the quoted
    self-referential forward-reference convention (the file has no
    from __future__ import annotations, and the class name is not yet
    bound in scope while its own body is being defined). This mirrors the
    identical convention already used on the sibling
    CharBPETokenizer.from_file -> "CharBPETokenizer" pattern for the same
    situation elsewhere in implementations/.
  • train and train_from_iterator both call self._tokenizer.train(...) /
    self._tokenizer.train_from_iterator(...) without a return statement,
    so they implicitly return None — hence -> None.

Scope note

Ran gh pr list --repo huggingface/tokenizers --search "byte_level_bpe" and
--search "ByteLevelBPETokenizer" before starting; no open or closed PR
currently touches this file. Re-checked immediately before opening this PR.

No behaviour change

Type-hint-only change on method signatures. No logic, docstrings, or other
files were modified.

Testing

  • ruff check bindings/python/py_src/tokenizers/implementations/byte_level_bpe.py — all checks passed.
  • ruff format --check bindings/python/py_src/tokenizers/implementations/byte_level_bpe.py — passes, file already formatted.
  • python3 -c "import ast; ast.parse(...)" on the modified file — passes.

Note: Claude Code was used to assist in drafting this change. All changes were reviewed by the submitter.

@github-actions github-actions Bot added the Stale label Sep 18, 2026
@github-actions github-actions Bot closed this Sep 29, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant