tiktoken-rs

Text tokenizer library

Provides a Rust library for tokenizing text with OpenAI models using tiktoken.

Ready-made tokenizer library for working with GPT and tiktoken

GitHub
256 stars
5 watching
47 forks
Language: Rust
last commit: 9 days ago
Linked from 1 awesome list

bpeopenairusttokenizer

Backlinks from these awesome lists:

Related projects:

Repository Description Stars
arbox/tokenizer A Ruby-based library for splitting written text into tokens for natural language processing tasks. 46
zseder/huntoken A tool for tokenizing raw text into words and sentences in multiple languages. 3
zencephalon/tactful_tokenizer A Ruby library that tokenizes text into sentences using a Bayesian statistical model 80
jonsafari/tok-tok A fast and simple tokenizer for multiple languages 28
mathewsanders/mustard A Swift library for tokenizing strings with customizable matching behavior 689
thisiscetin/textoken A gem for extracting words from text with customizable tokenization rules 31
jirkamarsik/trainable-tokenizer A tool for creating customizable tokenization rules for natural languages 22
fltk-rs/fltk-rs Rust bindings for a lightweight GUI library with customizable widgets and permissive licensing. 1,614
diasks2/pragmatic_tokenizer A multilingual tokenizer to split strings into tokens, handling various language and formatting nuances. 90
languagemachines/ucto A tokeniser for natural language text that separates words from punctuation and supports basic preprocessing steps such as case changing 65
6/tiny_segmenter A Ruby port of a Japanese text tokenization algorithm 21
nytud/quntoken A C++ tokenizer that tokenizes Hungarian text 14
amir-zeldes/rftokenizer A tokenizer for segmenting words into morphological components 27
abitdodgy/words_counted A Ruby library that tokenizes input and provides various statistical measures about the tokens 159
rkcosmos/deepcut A Thai word tokenization library using Deep Neural Network 420