WordTokenizers.jl
Tokenizer
A set of high-performance tokenizers for natural language processing tasks
High performance tokenizers for natural language processing and other related tasks
96 stars
9 watching
25 forks
Language: Julia
last commit: almost 3 years ago
Linked from 2 awesome lists
data-mininginformation-retrievallexernlptokenization
josephmisiti/awesome-machine-learning
keon/awesome-nlpRelated projects:
| Repository | Description | Stars |
|---|---|---|
 diasks2/pragmatic_tokenizer | A multilingual tokenizer to split strings into tokens, handling various language and formatting nuances. | 90 |
 thisiscetin/textoken | A gem for extracting words from text with customizable tokenization rules | 31 |
 arbox/tokenizer | A Ruby-based library for splitting written text into tokens for natural language processing tasks. | 46 |
 shonfeder/tokenize | A Prolog-based tokenization library for lexing text into common tokens | 11 |
 abitdodgy/words_counted | A Ruby library that tokenizes input and provides various statistical measures about the tokens | 159 |
 bzick/tokenizer | A high-performance tokenization library for Go, capable of parsing various data formats and syntaxes. | 101 |
 languagemachines/ucto | A tokeniser for natural language text that separates words from punctuation and supports basic preprocessing steps such as case changing | 65 |
 denosaurs/tokenizer | A simple tokenizer library for parsing and analyzing text input in various formats. | 17 |
 zseder/huntoken | A shell-based tool for breaking down raw text into words and sentences | 3 |
 jonsafari/tok-tok | A fast and simple tokenizer for multiple languages | 28 |
 c4n/pythonlexto | A Python wrapper around the Thai word segmentator LexTo, allowing developers to easily integrate it into their applications. | 1 |
 mathewsanders/mustard | A Swift library for tokenizing strings with customizable matching behavior | 689 |
 neurosnap/sentences | A command line tool to split text into individual sentences | 440 |
 zencephalon/tactful_tokenizer | A Ruby library that tokenizes text into sentences using a Bayesian statistical model | 80 |
| juliatext/languages.jl | A package for working with human languages in Julia. | 54 |