minbpe
by karpathy
Minimal, clean code for the Byte Pair Encoding (BPE) algorithm commonly used in LLM tokenization.
AI summary
BPE algorithm
An implementation of the Byte Pair Encoding algorithm used in language model tokenization.
- stars
- 9.3K
- forks
- 871
- watching
- 85
- awesome list
- 1
Similar projects
Found by comparing what the projects do, not just their names.
openai/tiktoken12.7K
Tokeniser
A fast and efficient tokeniser for natural language models based on Byte Pair Encoding (BPE)
Tokenizer toolkit
A toolkit providing optimized tokenizers for natural language processing tasks in various programming languages.
PEG parser library
A minimalist C++20 library for building pattern-matchers and parsers using Parsing Expression Grammars (PEGs)
ddbourgin/numpy-ml15.8K
ML toolkit
A collection of machine learning algorithms implemented in NumPy for rapid experimentation and prototyping.
Transformer model
A minimal PyTorch implementation of a transformer-based language model
Sentence tokenizer
A Ruby port of the NLTK algorithm to detect sentence boundaries in unstructured text
babel/minify4.4K
JavaScript minimizer
A tool that uses Babel's compiler to achieve minification of modern JavaScript code by targeting only browsers that support newer ES features.
Sentence embedder
An open source framework for learning sentence embeddings using contrastive learning.
bytedance/byteps3.6K
Distributed DL framework
A high-performance distributed deep learning framework supporting multiple frameworks and networks
Boolean solver
An implementation of Grover's algorithm using Qiskit and a large language model to generate boolean expressions from narratives
Byte serializer
A C++ serialization library that efficiently packs and unpacks structured data into compact byte arrays.
thunlp/plmpapers3.3K
Language model research paper collection
Compiles and organizes key papers on pre-trained language models, providing a resource for developers and researchers.
google/sentencepiece10.4K
Text segmenter
An unsupervised text tokenizer that segments input text into subwords and detokenizes output based on a predefined vocabulary size.
ByteString serializer
Efficient serialisation of values to and from lazy ByteStrings in Haskell.
aksnzhy/xlearn3.1K
Machine Learning Library
A high-performance machine learning package with linear models and factorization machines.
Featured in 1 awesome list
Each link jumps to the spot where the list mentions minbpe.
