sentencepiece
by google
Unsupervised text tokenizer for Neural Network-based text generation.
AI summary
Text segmenter
An unsupervised text tokenizer that segments input text into subwords and detokenizes output based on a predefined vocabulary size.
- stars
- 10.4K
- forks
- 1.2K
- watching
- 126
Similar projects
Found by comparing what the projects do, not just their names.
Tokenizer toolkit
A toolkit providing optimized tokenizers for natural language processing tasks in various programming languages.
Text tokenizer
A tokeniser for natural language text that separates words from punctuation and supports basic preprocessing steps such as case changing
Prompt Generator
A toolkit for creating and using natural language prompts to enable large language models to generalize to new tasks.
Sentence tokenizer
A command line tool to split text into individual sentences
Text Generator
A Python module for creating character-level or word-level neural networks for text generation and training on various datasets
NLP toolkit
An NLP project offering various text classification models and techniques for deep learning exploration
NLP toolkit
A Python library for natural language processing tasks in many human languages.
Word Vector Library
Provides pre-trained word vector representations and an implementation of the GloVe model for learning word embeddings
Text Generation Toolkit
A toolkit for deploying and serving Large Language Models (LLMs) for high-performance text generation
Sentence embedder
An open source framework for learning sentence embeddings using contrastive learning.
NLP course materials
An open-source repository containing lecture slides and course materials for an advanced natural language processing course.
Sentence tokenizer
A Ruby port of the NLTK algorithm to detect sentence boundaries in unstructured text
Sentence tokenizer
A Ruby library that tokenizes text into sentences using a Bayesian statistical model
Code completion tool
A code completion model trained on large amounts of programming language data to help developers write code more efficiently.
Captioning model
Efficient image captioning model using a CNN followed by an RNN in deep learning on GPU
