tokenizer
Text parser
A simple tokenizer library for parsing and analyzing text input in various formats.
⚙️ A simple tokenizer for deno
17 stars
3 watching
3 forks
Language: TypeScript
last commit: over 3 years ago
Linked from 1 awesome list
denolexertokenizer
Related projects:
| Repository | Description | Stars |
|---|---|---|
 arbox/tokenizer | A Ruby-based library for splitting written text into tokens for natural language processing tasks. | 46 |
 zseder/huntoken | A tool for tokenizing raw text into words and sentences in multiple languages, including Hungarian. | 4 |
 bzick/tokenizer | A high-performance tokenization library for Go, capable of parsing various data formats and syntaxes. | 103 |
 diasks2/pragmatic_tokenizer | A multilingual tokenizer to split strings into tokens, handling various language and formatting nuances. | 90 |
 siokas/denomander | A Deno-based framework for building command-line interfaces | 150 |
 juliatext/wordtokenizers.jl | A set of high-performance tokenizers for natural language processing tasks | 96 |
 shonfeder/tokenize | A Prolog-based tokenization library for lexing text into common tokens | 11 |
 denolib/marked | A TypeScript port of a popular markdown parser with type annotations for use in Deno applications. | 9 |
 neurosnap/sentences | A command line tool to split text into individual sentences | 441 |
 thisiscetin/textoken | A gem for extracting words from text with customizable tokenization rules | 31 |
 6/tiny_segmenter | A Ruby port of a Japanese text tokenization algorithm | 21 |
 jonsafari/tok-tok | A fast and simple tokenizer for multiple languages | 28 |
 mathewsanders/mustard | A Swift library for tokenizing strings with customizable matching behavior | 689 |
 zencephalon/tactful_tokenizer | A Ruby library that tokenizes text into sentences using a Bayesian statistical model | 80 |
| denosaurs/status | A utility for working with HTTP status codes in Deno web applications. | 15 |