quntoken
Hungarian tokenizer
A C++ tokenizer that tokenizes Hungarian text
Hungarian tokenizer.
14 stars
15 watching
5 forks
Language: C++
last commit: over 4 years agoLinked from 1 awesome list
oroszgy/awesome-hungarian-nlpRelated projects:
| Repository | Description | Stars |
|---|---|---|
| nytud/hunlp-gate | A collection of Hungarian NLP tools integrated as GATE processing resources | 8 |
| nytud/hucola | A collection of 9,076 annotated sentences in Hungarian to evaluate linguistic acceptability and grammaticality | 1 |
 languagemachines/ucto | A tokeniser for natural language text that separates words from punctuation and supports basic preprocessing steps such as case changing | 66 |
 jonsafari/tok-tok | A fast and simple tokenizer for multiple languages | 28 |
| nytud/panmorph | Harmonized tagset and annotation scheme for Hungarian morphological analysers | 4 |
 zseder/huntoken | A tool for tokenizing raw text into words and sentences in multiple languages, including Hungarian. | 4 |
| nytud/emtsv | A text processing system designed to handle various tasks in Hungarian language processing using Python and TSV-based data exchange. | 28 |
| nytud/nytk-nerkor | A Hungarian language named entity annotated corpus containing 1 million tokens with morphological annotation layers and various source files. | 15 |
| nytud/emlam | Preprocessing and modeling scripts for Hungarian language modeling using Python and TensorFlow. | 3 |
 zurawiki/tiktoken-rs | Provides a Rust library for tokenizing text with OpenAI models using tiktoken. | 266 |
 bzick/tokenizer | A high-performance tokenization library for Go, capable of parsing various data formats and syntaxes. | 103 |
 shonfeder/tokenize | A Prolog-based tokenization library for lexing text into common tokens | 11 |
 thisiscetin/textoken | A gem for extracting words from text with customizable tokenization rules | 31 |
 jirkamarsik/trainable-tokenizer | A tool for creating customizable tokenization rules for natural languages | 22 |
 diasks2/pragmatic_tokenizer | A multilingual tokenizer to split strings into tokens, handling various language and formatting nuances. | 90 |