quntoken
by nytud
Hungarian tokenizer.
AI summary
Hungarian tokenizer
A C++ tokenizer that tokenizes Hungarian text
- stars
- 14
- forks
- 5
- watching
- 15
- awesome list
- 1
Similar projects
Found by comparing what the projects do, not just their names.
NLP toolkit
A collection of Hungarian NLP tools integrated as GATE processing resources
Hungarian Corpus
A collection of 9,076 annotated sentences in Hungarian to evaluate linguistic acceptability and grammaticality
Text tokenizer
A tokeniser for natural language text that separates words from punctuation and supports basic preprocessing steps such as case changing
Tokenizer
A fast and simple tokenizer for multiple languages
Hungarian tagset
Harmonized tagset and annotation scheme for Hungarian morphological analysers
Text tokenizer
A tool for tokenizing raw text into words and sentences in multiple languages, including Hungarian.
Hungarian NLP system
A text processing system designed to handle various tasks in Hungarian language processing using Python and TSV-based data exchange.
Hungarian NE corpus
A Hungarian language named entity annotated corpus containing 1 million tokens with morphological annotation layers and various source files.
Language model script
Preprocessing and modeling scripts for Hungarian language modeling using Python and TensorFlow.
Text tokenizer library
Provides a Rust library for tokenizing text with OpenAI models using tiktoken.
Tokenizer
A high-performance tokenization library for Go, capable of parsing various data formats and syntaxes.
Tokenizer
A Prolog-based tokenization library for lexing text into common tokens
Tokenizer
A gem for extracting words from text with customizable tokenization rules
Tokenizer builder
A tool for creating customizable tokenization rules for natural languages
Tokenizer
A multilingual tokenizer to split strings into tokens, handling various language and formatting nuances.
Featured in 1 awesome list
Each link jumps to the spot where the list mentions quntoken.
