Package: python3-tokenizers
Source: tokenizers
Version: 0.23.1+dfsg-1
Architecture: amd64
Maintainer: Ubuntu Developers <ubuntu-devel-discuss@lists.ubuntu.com>
Original-Maintainer: Debian Deep Learning Team <debian-ai@lists.debian.org>
Installed-Size: 6095
Depends: python3, python3-huggingface-hub, python3:any, libc6 (>= 2.34), libgcc-s1 (>= 4.2), libonig5 (>= 6.8.1), libstdc++6 (>= 4.1.1)
Section: python
Priority: optional
Homepage: https://github.com/huggingface/tokenizers
Description: Fast State-of-the-Art Tokenizers for Research and Production
 Provides an implementation of today's most used tokenizers, with a focus on
 performance and versatility.
 .
 Main features:
  * Train new vocabularies and tokenize, using today's most used tokenizers.
  * Extremely fast (both training and tokenization), thanks to the Rust
    implementation. Takes less than 20 seconds to tokenize a GB of text on
    a server's CPU.
  * Easy to use, but also extremely versatile.
  * Designed for research and production.
  * Normalization comes with alignments tracking. It's always possible to get
    the part of the original sentence that corresponds to a given token.
  * Does all the pre-processing: Truncate, Pad, add the special tokens your
    model needs.
