tokenizer
by bzick
Tokenizer (lexer) for golang
AI summary
Tokenizer
A high-performance tokenization library for Go, capable of parsing various data formats and syntaxes.
- stars
- 103
- forks
- 7
- watching
- 2
- awesome lists
- 2
Similar projects
Found by comparing what the projects do, not just their names.
Tokenizer
A multilingual tokenizer to split strings into tokens, handling various language and formatting nuances.
Chinese Tokenizer Library
A tokenizer based on dictionary and Bigram language models for text segmentation in Chinese
Tokenizer
A Prolog-based tokenization library for lexing text into common tokens
Text parser
A simple tokenizer library for parsing and analyzing text input in various formats.
Tokenizer
A fast and simple tokenizer for multiple languages
Tokenizer
A set of high-performance tokenizers for natural language processing tasks
Text tokenizer library
A Ruby-based library for splitting written text into tokens for natural language processing tasks.
XML tokenizer
A library that tokenizes XML data into smaller units for easier processing
Text tokenizer
A tool for tokenizing raw text into words and sentences in multiple languages, including Hungarian.
Tokenizer
A Ruby library that tokenizes input and provides various statistical measures about the tokens
Tokenizer
A gem for extracting words from text with customizable tokenization rules
Sentence tokenizer
A command line tool to split text into individual sentences
CSS tokenizer
A utility for parsing and breaking down CSS3 code into smaller components
Parser generator
A tool for generating lexers and parsers from a BNF file with semantic actions.
Tokenizer builder
A tool for creating customizable tokenization rules for natural languages
Featured in 2 awesome lists
Each link jumps to the spot where the list mentions tokenizer.
