segtok
by fnl
Segtok v2 is here: https://github.com/fnl/syntok -- A rule-based sentence segmenter (splitter) and a word tokenizer using orthographic features.
AI summary
Sentence splitter
Provides tools for splitting text into sentences and words
- stars
- 171
- forks
- 22
- watching
- 11
Similar projects
Found by comparing what the projects do, not just their names.
Word tokenizer
A Python wrapper around the Thai word segmentator LexTo, allowing developers to easily integrate it into their applications.
Sentence embedder
A method to generate sentence embeddings from pre-trained language models
Sentence tokenizer
A command line tool to split text into individual sentences
Term snippet
Adds support for glossary terms entity to Draftail in Wagtail
Text splitter
A Common Lisp library for splitting text into manageable segments based on document structure and layout characteristics.
Prompt decorator
Provides syntactic sugar for writing custom LangChain prompts and chains, making it easier to write more pythonic code.
Sentence tokenizer
A Ruby port of the NLTK algorithm to detect sentence boundaries in unstructured text
Tokenizer
A fast and simple tokenizer for multiple languages
Sentence evaluation tool
Tool for evaluating the quality of sentence embeddings as features in various downstream tasks.
Thai word segementation tool
A tool for segmenting Thai text into words using Recurrent Neural Networks in TensorFlow.
Syntax Highlighter
A Python syntax highlighter package for multiple text editors.
Sentence splitter
A Ruby library that uses a simple rule-based approach to segment sentences into individual words or phrases.
Log parser library
Provides parsers for parsing Windows log file formats
Prolog parser
Provides a Prolog grammar and parser for tree-sitter, enabling parsing of various Prolog formats.
Sentence embedding toolkit
A tool for training and evaluating sentence embeddings using deep contrastive learning
