tiktoken-rs
by zurawiki
Ready-made tokenizer library for working with GPT and tiktoken
AI summary
Text tokenizer library
Provides a Rust library for tokenizing text with OpenAI models using tiktoken.
- stars
- 266
- forks
- 49
- watching
- 5
- awesome list
- 1
Similar projects
Found by comparing what the projects do, not just their names.
Text tokenizer library
A Ruby-based library for splitting written text into tokens for natural language processing tasks.
Text tokenizer
A tool for tokenizing raw text into words and sentences in multiple languages, including Hungarian.
Sentence tokenizer
A Ruby library that tokenizes text into sentences using a Bayesian statistical model
Tokenizer
A fast and simple tokenizer for multiple languages
tokenizer library
A Swift library for tokenizing strings with customizable matching behavior
Tokenizer
A gem for extracting words from text with customizable tokenization rules
Tokenizer builder
A tool for creating customizable tokenization rules for natural languages
fltk-rs/fltk-rs1.6K
GUI library
Rust bindings for a cross-platform GUI library
Tokenizer
A multilingual tokenizer to split strings into tokens, handling various language and formatting nuances.
Text tokenizer
A tokeniser for natural language text that separates words from punctuation and supports basic preprocessing steps such as case changing
Text tokenizer
A Ruby port of a Japanese text tokenization algorithm
Hungarian tokenizer
A C++ tokenizer that tokenizes Hungarian text
Morph seg tokenizer
A tokenizer for segmenting words into morphological components
Tokenizer
A Ruby library that tokenizes input and provides various statistical measures about the tokens
Thai text tokenizer
A Thai word tokenization library using Deep Neural Network
Featured in 1 awesome list
Each link jumps to the spot where the list mentions tiktoken-rs.
