ucto
Unicode tokeniser. Ucto tokenizes text files: it separates words from punctuation, and splits sentences. It offers several other basic preprocessing steps such as changing case that you can all use to make your text suited for further processing such as indexing, part-of-speech tagging, or machine translation. Ucto comes with tokenisation rules for several languages and can be easily extended to suit other languages. It has been incorporated for tokenizing Dutch text in Frog, our Dutch morpho-syntactic processor. http://ilk.uvt.nl/ucto --
AI summary
Text tokenizer
A tokeniser for natural language text that separates words from punctuation and supports basic preprocessing steps such as case changing
- stars
- 66
- forks
- 13
- watching
- 13
- awesome lists
- 2
Similar projects
Found by comparing what the projects do, not just their names.
Tokeniser library
A Python binding to an advanced, extensible tokeniser written in C++
Word tokenizer
A Python wrapper around the Thai word segmentator LexTo, allowing developers to easily integrate it into their applications.
Tokenizer
A set of high-performance tokenizers for natural language processing tasks
Text tokenizer library
A Ruby-based library for splitting written text into tokens for natural language processing tasks.
Hungarian tokenizer
A C++ tokenizer that tokenizes Hungarian text
Tokenizer
A gem for extracting words from text with customizable tokenization rules
Text parser
A simple tokenizer library for parsing and analyzing text input in various formats.
Tokenizer
A fast and simple tokenizer for multiple languages
Text tokenizer
A tool for tokenizing raw text into words and sentences in multiple languages, including Hungarian.
Sentence tokenizer
A command line tool to split text into individual sentences
Tokenizer
A multilingual tokenizer to split strings into tokens, handling various language and formatting nuances.
Sentence tokenizer
A Ruby port of the NLTK algorithm to detect sentence boundaries in unstructured text
Sentence tokenizer
A Ruby library that tokenizes text into sentences using a Bayesian statistical model
Text tokenizer library
Provides a Rust library for tokenizing text with OpenAI models using tiktoken.
Tokenizer
A Prolog-based tokenization library for lexing text into common tokens
Featured in 2 awesome lists
Each link jumps to the spot where the list mentions ucto.
