tiktoken
by openai
tiktoken is a fast BPE tokeniser for use with OpenAI's models.
AI summary
Tokeniser
A fast and efficient tokeniser for natural language models based on Byte Pair Encoding (BPE)
- stars
- 12.7K
- forks
- 873
- watching
- 169
Similar projects
Found by comparing what the projects do, not just their names.
Tokenizer toolkit
A toolkit providing optimized tokenizers for natural language processing tasks in various programming languages.
Text tokenizer library
Provides a Rust library for tokenizing text with OpenAI models using tiktoken.
RL frameworks
High-quality implementations of reinforcement learning algorithms for research and development purposes
karpathy/minbpe9.3K
BPE algorithm
An implementation of the Byte Pair Encoding algorithm used in language model tokenization.
Code generator
An interactive code generation and execution tool using AI models
Language model repository
A repository providing code and models for research into language modeling and multitask learning
Speech Recogniser
A general-purpose speech recognition system trained on large-scale weak supervision
openbmb/bmtools2.9K
LM toolkit
Tools and platform for building and extending large language models
Image generator
This project is a software implementation of a diffusion model architecture, allowing users to generate synthetic images based on a learned distribution.
Prompt engineering guide
Guides software developers on how to effectively use and build systems around Large Language Models like GPT-4.
AI model trainer
An open-source toolkit for training and deploying large-scale AI models on various downstream tasks with multi-modality
API wrapper
Provides client-side access to ChatGPT and Bing AI APIs using Node.js
Model debugger
An open-source tool that helps investigate specific behaviors of small language models by combining automated interpretability techniques with sparse autoencoders.
Text generator
A tool for retraining and fine-tuning the OpenAI GPT-2 text generation model on new datasets.
OpenAI client
A PHP SDK for accessing the OpenAI API and interacting with its GPT-3 and DALL-E services.
