tokenizers
by huggingface
💥 Fast State-of-the-Art Tokenizers optimized for Research and Production
AI summary
Tokenizer toolkit
A toolkit providing optimized tokenizers for natural language processing tasks in various programming languages.
- stars
- 9.2K
- forks
- 815
- watching
- 122
- awesome lists
- 3
Similar projects
Found by comparing what the projects do, not just their names.
openai/tiktoken12.7K
Tokeniser
A fast and efficient tokeniser for natural language models based on Byte Pair Encoding (BPE)
google/sentencepiece10.4K
Text segmenter
An unsupervised text tokenizer that segments input text into subwords and detokenizes output based on a predefined vocabulary size.
Text Generation Toolkit
A toolkit for deploying and serving Large Language Models (LLMs) for high-performance text generation
ML data loader
A tool providing efficient data manipulation and loading for machine learning models
Model runner
An open-source JavaScript library for running machine learning models in the browser without a server.
Transformer model trainer
A library designed to train transformer language models with reinforcement learning using various optimization techniques and fine-tuning methods.
Text tokenizer library
A Ruby-based library for splitting written text into tokens for natural language processing tasks.
Text tokenizer
A tokeniser for natural language text that separates words from punctuation and supports basic preprocessing steps such as case changing
karpathy/minbpe9.3K
BPE algorithm
An implementation of the Byte Pair Encoding algorithm used in language model tokenization.
Tokenizer
A fast and simple tokenizer for multiple languages
Parameter adaptation
An efficient method for fine-tuning large pre-trained models by adapting only a small fraction of their parameters
Model repository
A collection of pre-trained machine learning models for various natural language and computer vision tasks, enabling developers to fine-tune and deploy these models on their own projects.
Tokeniser library
A Python binding to an advanced, extensible tokeniser written in C++
Sentence tokenizer
A Ruby library that tokenizes text into sentences using a Bayesian statistical model
Alignment Recipes
Provides recipes and guidelines for training language models to align with human preferences and AI goals
Featured in 3 awesome lists
Each link jumps to the spot where the list mentions tokenizers.
