tools · huggingface

Tokenizers

Rust-backed BPE/WordPiece/Unigram tokenizer with Python bindings. Sub-millisecond encoding for production pipelines.

Verify

What this registry has checked

Quarantine. We hold the publisher’s own published hash for these weights and have recorded it against this entry. We have not downloaded and hashed the weights ourselves, so the hash is the publisher’s claim, not our measurement.

Weight file SHA-256	724d1d4672321f3ed5e2fc58e66f61d7210508326c813c2f8b2b9fcb9bedf893

Re-check this yourself → Source → Full record →

What is recorded

Publisher	huggingface
Origin	United States
Licence	Apache-2.0
Pulls recorded here	134,001

Elsewhere in the registry

All tools → Browse with filters → What will run on my hardware? →

Registry record as of 2026-06-06. Identity, licence and parameter count are properties of a release and do not move; tier can, and the verify link above re-checks it against the live registry rather than this page.