tiktoken-rs
Text tokenizer library
Provides a Rust library for tokenizing text with OpenAI models using tiktoken.
Ready-made tokenizer library for working with GPT and tiktoken
266 stars
5 watching
49 forks
Language: Rust
last commit: 12 months ago
Linked from 1 awesome list
bpeopenairusttokenizer
rust-unofficial/awesome-rustRelated projects:
| Repository | Description | Stars |
|---|---|---|
 arbox/tokenizer | A Ruby-based library for splitting written text into tokens for natural language processing tasks. | 46 |
 zseder/huntoken | A tool for tokenizing raw text into words and sentences in multiple languages, including Hungarian. | 4 |
 zencephalon/tactful_tokenizer | A Ruby library that tokenizes text into sentences using a Bayesian statistical model | 80 |
 jonsafari/tok-tok | A fast and simple tokenizer for multiple languages | 28 |
 mathewsanders/mustard | A Swift library for tokenizing strings with customizable matching behavior | 689 |
 thisiscetin/textoken | A gem for extracting words from text with customizable tokenization rules | 31 |
 jirkamarsik/trainable-tokenizer | A tool for creating customizable tokenization rules for natural languages | 22 |
 fltk-rs/fltk-rs | Rust bindings for a cross-platform GUI library | 1,646 |
 diasks2/pragmatic_tokenizer | A multilingual tokenizer to split strings into tokens, handling various language and formatting nuances. | 90 |
 languagemachines/ucto | A tokeniser for natural language text that separates words from punctuation and supports basic preprocessing steps such as case changing | 66 |
 6/tiny_segmenter | A Ruby port of a Japanese text tokenization algorithm | 21 |
 nytud/quntoken | A C++ tokenizer that tokenizes Hungarian text | 14 |
 amir-zeldes/rftokenizer | A tokenizer for segmenting words into morphological components | 27 |
 abitdodgy/words_counted | A Ruby library that tokenizes input and provides various statistical measures about the tokens | 159 |
 rkcosmos/deepcut | A Thai word tokenization library using Deep Neural Network | 421 |