gotokenizer
by xujiajun
A tokenizer based on the dictionary and Bigram language models for Go. (Now only support chinese segmentation)
AI summary
Chinese Tokenizer Library
A tokenizer based on dictionary and Bigram language models for text segmentation in Chinese
- stars
- 21
- forks
- 7
- watching
- 3
- awesome lists
- 2
Similar projects
Found by comparing what the projects do, not just their names.
Tokenizer
A high-performance tokenization library for Go, capable of parsing various data formats and syntaxes.
Chinese Segmentation Library
A Python library for Chinese text segmentation using a Hidden Makov Model algorithm
Tokenizer
A fast and simple tokenizer for multiple languages
Tokenizer
A gem for extracting words from text with customizable tokenization rules
Router
A fast and feature-rich HTTP router for Go that supports regular expressions.
Chinese tokenizer
A PHP module for Chinese text segmentation and word breaking
Chinese tokenizer library
Provides a Ruby port of the popular Chinese language processing library Jieba
Sentence tokenizer
A Ruby library that tokenizes text into sentences using a Bayesian statistical model
Design pattern guide
A comprehensive guide to design patterns in Go programming language
Tokenizer
A multilingual tokenizer to split strings into tokens, handling various language and formatting nuances.
Text tokenizer
A Ruby port of a Japanese text tokenization algorithm
Tokenizer
A Ruby library that tokenizes input and provides various statistical measures about the tokens
Text tokenizer
A tool for tokenizing raw text into words and sentences in multiple languages, including Hungarian.
Text tokenizer library
A Ruby-based library for splitting written text into tokens for natural language processing tasks.
Functional programming language
A Go implementation of Shen, a portable functional programming language with features like pattern matching and macro support.
Featured in 2 awesome lists
Each link jumps to the spot where the list mentions gotokenizer.
