python-ucto
by proycon
This is a Python binding to the tokenizer Ucto. Tokenisation is one of the first step in almost any Natural Language Processing task, yet it is not always as trivial a task as it appears to be. This binding makes the power of the ucto tokeniser available to Python. Ucto itself is regular-expression based, extensible, and advanced tokeniser written in C++ (http://ilk.uvt.nl/ucto).
AI summary
Tokeniser library
A Python binding to an advanced, extensible tokeniser written in C++
- stars
- 29
- forks
- 5
- watching
- 4
- awesome lists
- 2
Similar projects
Found by comparing what the projects do, not just their names.
Text tokenizer
A tokeniser for natural language text that separates words from punctuation and supports basic preprocessing steps such as case changing
Word tokenizer
A Python wrapper around the Thai word segmentator LexTo, allowing developers to easily integrate it into their applications.
Tokenizer
A fast and simple tokenizer for multiple languages
NLP library
A Python binding to a C++ NLP tool for Dutch language processing tasks
Text tokenizer library
A Ruby-based library for splitting written text into tokens for natural language processing tasks.
Tokenizer
A Prolog-based tokenization library for lexing text into common tokens
Sentence tokenizer
A Ruby port of the NLTK algorithm to detect sentence boundaries in unstructured text
Text generator
Develops pretraining and finetuning techniques for language models using metadata-conditioned text generation
Hungarian tokenizer
A C++ tokenizer that tokenizes Hungarian text
Parser combinator library
A header-only C++ library for defining and implementing parser combinators using a Parsing Expression Grammar approach
Thai text tokenizer
A Thai word tokenization library using Deep Neural Network
Tokenizer
A gem for extracting words from text with customizable tokenization rules
Tokenizer
A Ruby library that tokenizes input and provides various statistical measures about the tokens
Text parser
A simple tokenizer library for parsing and analyzing text input in various formats.
Python standard library
Develops a minimal and lightweight Python standard library compatible with other variants and implementations of Python.
Featured in 2 awesome lists
Each link jumps to the spot where the list mentions python-ucto.
