awesome-nlp
A curated list of resources dedicated to Natural Language Processing (NLP)
17k stars
610 watching
3k forks
last commit: 11 months ago
Linked from 8 awesome lists
awesomeawesome-listdeep-learninglanguagemachine-learningnatural-language-processingnlptext-mining
awesome-nlp / Research Summaries and Trends | |||
NLP-Overview | is an up-to-date overview of deep learning techniques applied to NLP, including theory, implementations, applications, and state-of-the-art results. This is a great Deep NLP Introduction for researchers | ||
NLP-Progress | tracks the progress in Natural Language Processing, including the datasets and the current state-of-the-art for the most common NLP tasks | ||
NLP's ImageNet moment has arrived | |||
ACL 2018 Highlights: Understanding Representation and Evaluation in More Challenging Settings | |||
Four deep learning trends from ACL 2017. Part One: Linguistic Structure and Word Embeddings | |||
Four deep learning trends from ACL 2017. Part Two: Interpretability and Attention | |||
Highlights of EMNLP 2017: Exciting Datasets, Return of the Clusters, and More! | |||
Deep Learning for Natural Language Processing (NLP): Advancements & Trends | |||
Survey of the State of the Art in Natural Language Generation | |||
awesome-nlp / Prominent NLP Research Labs | |||
The Berkeley NLP Group | Notable contributions include a tool to reconstruct long dead languages, referenced and by taking corpora from 637 languages currently spoken in Asia and the Pacific and recreating their descendant | ||
Language Technologies Institute, Carnegie Mellon University | Notable projects include , a syntax driven machine translation system for endangered languages like Quechua and Aymara and previously, which created to improve NLP tools for Arabic | ||
NLP research group, Columbia University | Responsible for creating BOLT ( interactive error handling for speech translation systems) and an un-named project to characterize laughter in dialogue | ||
The Center or Language and Speech Processing, John Hopkins University | Recently in the news for developing speech recognition software to create a diagnostic test or Parkinson's Disease, | ||
Computational Linguistics and Information Processing Group, University of Maryland | Notable contributions include and modeling development of phonetic representations | ||
Penn Natural Language Processing, University of Pennsylvania | Famous for creating the | ||
The Stanford Nautral Language Processing Group | One of the top NLP research labs in the world, notable for creating and their | ||
awesome-nlp / Tutorials / Reading Content | |||
Machine Learning 101 | from Google's Senior Creative Engineer explains Machine Learning for engineer's and executives alike | ||
AI Playbook | a16z AI playbook is a great link to forward to your managers or content for your presentations | ||
Ruder's Blog | by for commentary on the best of NLP Research | ||
How To Label Data | guide to managing larger linguistic annotation projects | ||
Depends on the Definition | collection of blog posts covering a wide array of NLP topics with detailed implementation | ||
Understand & Implement Natural Language Processing | |||
NLP in Python | Collection of Github notebooks | ||
Natural Language Processing: An Introduction | Oxford | ||
Deep Learning for NLP with Pytorch | |||
Hands-On NLTK Tutorial | 540 | 5 months ago | NLTK Tutorials, Jupyter notebooks |
Natural Language Processing with Python – Analyzing Text with the Natural Language Toolkit | An online and print book introducing NLP concepts using NLTK. The book's authors also wrote the NLTK library | ||
Train a new language model from scratch | Hugging Face 🤗 | ||
The Super Duper NLP Repo (SDNLPR) | : Collection of Colab notebooks covering a wide array of NLP task implementations | ||
Deep Learning, NLP, and Representations | |||
The Illustrated BERT, ELMo, and co. (How NLP Cracked Transfer Learning) | and | ||
Natural Language Processing | by Hal Daumé III | ||
arXiv: Natural Language Processing (Almost) from Scratch | |||
Karpathy's The Unreasonable Effectiveness of Recurrent Neural Networks | |||
Machine Learning Mastery: Deep Learning for Natural Language Processing | |||
Visual NLP Paper Summaries | |||
awesome-nlp / Tutorials / Videos and Online Courses | |||
Advanced Natural Language Processing | CS 685, UMass Amherst CS | ||
Deep Natural Language Processing | 15,644 | over 1 year ago | Lectures series from Oxford |
Deep Learning for Natural Language Processing (cs224-n) | Richard Socher and Christopher Manning's Stanford Course | ||
Neural Networks for NLP | Carnegie Mellon Language Technology Institute there | ||
Deep NLP Course | 9,754 | 2 days ago | by Yandex Data School, covering important ideas from text embedding to machine translation including sequence modeling, language models and so on |
fast.ai Code-First Intro to Natural Language Processing | This covers a blend of traditional NLP topics (including regex, SVD, naive bayes, tokenization) and recent neural network approaches (including RNNs, seq2seq, GRUs, and the Transformer), as well as addressing urgent ethical issues, such as bias and disinformation. Find the Jupyter Notebooks | ||
Machine Learning University - Accelerated Natural Language Processing | Lectures go from introduction to NLP and text processing to Recurrent Neural Networks and Transformers. Material can be found | ||
Applied Natural Language Processing | Lecture series from IIT Madras taking from the basics all the way to autoencoders and everything. The github notebooks for this course are also available | ||
awesome-nlp / Tutorials / Books | |||
Speech and Language Processing | free, by Prof. Dan Jurafsy | ||
Natural Language Processing | 4,936 | over 1 year ago | free, NLP notes by Dr. Jacob Eisenstein at GeorgiaTech |
NLP with PyTorch | 1,985 | over 1 year ago | Brian & Delip Rao |
Text Mining in R | |||
Natural Language Processing with Python | |||
Practical Natural Language Processing | |||
Natural Language Processing with Spark NLP | |||
Deep Learning for Natural Language Processing | by Stephan Raaijmakers | ||
Real-World Natural Language Processing | by Masato Hagiwara | ||
Natural Language Processing in Action, Second Edition | by Hobson Lane and Maria Dyshel | ||
awesome-nlp / Libraries | |||
Twitter-text | 3,072 | 6 months ago | A JavaScript implementation of Twitter's text processing library |
Knwl.js | 5,294 | about 1 year ago | A Natural Language Processor in JS |
Retext | 2,360 | 9 days ago | Extensible system for analyzing and manipulating natural language |
NLP Compromise | 11,433 | 2 days ago | Natural Language processing in the browser |
Natural | 10,583 | 2 months ago | general natural language facilities for node |
Poplar | 519 | almost 2 years ago | A web-based annotation tool for natural language processing (NLP) |
NLP.js | 6,237 | 4 months ago | An NLP library for building bots |
node-question-answering | 464 | about 1 year ago | Fast and production-ready question answering w/ DistilBERT in Node.js |
sentimental-onix | 3 | over 1 year ago | Sentiment models for spacy using onnx |
TextAttack | 2,923 | 3 months ago | Adversarial attacks, adversarial training, and data augmentation in NLP |
TextBlob | Providing a consistent API for diving into common natural language processing (NLP) tasks. Stands on the giant shoulders of and , and plays nicely with both | ||
spaCy | 29,804 | 11 days ago | Industrial strength NLP with Python and Cython |
Speedster | 8,378 | 3 months ago | Automatically apply SOTA optimization techniques to achieve the maximum inference speed-up on your hardware |
awesome-nlp / Libraries / Speedster | |||
textacy | 2,209 | about 1 year ago | Higher level NLP built on spaCy |
awesome-nlp / Libraries | |||
gensim | Python library to conduct unsupervised semantic modelling from plain text | ||
scattertext | 2,236 | 19 days ago | Python library to produce d3 visualizations of how language differs between corpora |
GluonNLP | 2,555 | about 1 year ago | A deep learning toolkit for NLP, built on MXNet/Gluon, for research prototyping and industrial deployment of state-of-the-art models on a wide range of NLP tasks |
AllenNLP | 11,741 | almost 2 years ago | An NLP research library, built on PyTorch, for developing state-of-the-art deep learning models on a wide variety of linguistic tasks |
PyTorch-NLP | 2,210 | over 1 year ago | NLP research toolkit designed to support rapid prototyping with better data loaders, word vector loaders, neural network layer representations, common NLP metrics such as BLEU |
Rosetta | 206 | almost 2 years ago | Text processing tools and wrappers (e.g. Vowpal Wabbit) |
PyNLPl | 479 | about 1 year ago | Python Natural Language Processing Library. General purpose NLP library for Python, handles some specific formats like ARPA language models, Moses phrasetables, GIZA++ alignments |
foliapy | 18 | 5 months ago | Python library for working with , an XML format for linguistic annotation |
PySS3 | 333 | about 1 year ago | Python package that implements a novel white-box machine learning model for text classification, called SS3. Since SS3 has the ability to visually explain its rationale, this package also comes with easy-to-use interactive visualizations tools ( ) |
jPTDP | 157 | over 5 years ago | A toolkit for joint part-of-speech (POS) tagging and dependency parsing. jPTDP provides pre-trained models for 40+ languages |
BigARTM | 661 | about 1 year ago | a fast library for topic modelling |
Snips NLU | 3,889 | over 1 year ago | A production ready library for intent parsing |
Chazutsu | 244 | almost 2 years ago | A library for downloading&parsing standard NLP research datasets |
Word Forms | 620 | over 3 years ago | Word forms can accurately generate all possible forms of an English word |
Multilingual Latent Dirichlet Allocation (LDA) | 82 | 3 months ago | A multilingual and extensible document clustering pipeline |
Natural Language Toolkit (NLTK) | A library containing a wide variety of NLP functionality, supporting over 50 corpora | ||
NLP Architect | 2,936 | almost 2 years ago | A library for exploring the state-of-the-art deep learning topologies and techniques for NLP and NLU |
Flair | 13,852 | 14 days ago | A very simple framework for state-of-the-art multilingual NLP built on PyTorch. Includes BERT, ELMo and Flair embeddings |
Kashgari | 2,389 | about 1 month ago | Simple, Keras-powered multilingual NLP framework, allows you to build your models in 5 minutes for named entity recognition (NER), part-of-speech tagging (PoS) and text classification tasks. Includes BERT and word2vec embedding |
FARM | 1,734 | 10 months ago | Fast & easy transfer learning for NLP. Harvesting language models for the industry. Focus on Question Answering |
Haystack | 17,024 | 4 days ago | End-to-end Python framework for building natural language search interfaces to data. Leverages Transformers and the State-of-the-Art of NLP. Supports DPR, Elasticsearch, HuggingFace’s Modelhub, and much more! |
Rita DSL | 65 | about 2 years ago | a DSL, loosely based on . Allows to define language patterns (rule-based NLP) which are then translated into , or if you prefer less features and lightweight - regex patterns |
Transformers | 133,213 | 1 day ago | Natural Language Processing for TensorFlow 2.0 and PyTorch |
Tokenizers | 8,926 | 11 days ago | Tokenizers optimized for Research and Production |
fairSeq | 30,266 | 9 days ago | Facebook AI Research implementations of SOTA seq2seq models in Pytorch |
corex_topic | 626 | over 3 years ago | Hierarchical Topic Modeling with Minimal Domain Knowledge |
Sockeye | 1,210 | 4 months ago | Neural Machine Translation (NMT) toolkit that powers Amazon Translate |
DL Translate | 443 | about 1 month ago | A deep learning-based translation library for 50 languages, built on and Facebook's mBART Large |
Jury | 186 | 2 months ago | Evaluation of NLP model outputs offering various automated metrics |
python-ucto | 29 | about 1 month ago | Unicode-aware regular-expression based tokenizer for various languages. Python binding to C++ library, supports |
InsNet | 66 | almost 3 years ago | A neural network library for building instance-dependent NLP models with padding-free dynamic batching |
MIT Information Extraction Toolkit | 2,918 | about 2 years ago | C, C++, and Python tools for named entity recognition and relation extraction |
CRF++ | Open source implementation of Conditional Random Fields (CRFs) for segmenting/labeling sequential data & other Natural Language Processing tasks | ||
CRFsuite | CRFsuite is an implementation of Conditional Random Fields (CRFs) for labeling sequential data | ||
BLLIP Parser | 227 | almost 3 years ago | BLLIP Natural Language Parser (also known as the Charniak-Johnson parser) |
colibri-core | 123 | 11 months ago | C++ library, command line tools, and Python binding for extracting and working with basic linguistic constructions such as n-grams and skipgrams in a quick and memory-efficient way |
ucto | 65 | 18 days ago | Unicode-aware regular-expression based tokenizer for various languages. Tool and C++ library. Supports FoLiA format |
libfolia | 15 | 18 days ago | C++ library for the |
frog | 73 | 16 days ago | Memory-based NLP suite developed for Dutch: PoS tagger, lemmatiser, dependency parser, NER, shallow parser, morphological analyzer |
MeTA | 693 | over 1 year ago | is a C++ Data Sciences Toolkit that facilitates mining big text data |
Mecab (Japanese) | |||
Moses | |||
StarSpace | 3,938 | almost 2 years ago | a library from Facebook for creating embeddings of word-level, paragraph-level, document-level and for text classification |
Stanford NLP | |||
OpenNLP | |||
NLP4J | |||
Word2vec in Java | |||
ReVerb | 543 | over 5 years ago | Web-Scale Open Information Extraction |
OpenRegex | 75 | over 10 years ago | An efficient and flexible token-based regular expression language and engine |
CogcompNLP | 470 | over 1 year ago | Core libraries developed in the U of Illinois' Cognitive Computation Group |
MALLET | MAchine Learning for LanguagE Toolkit - package for statistical natural language processing, document classification, clustering, topic modeling, information extraction, and other machine learning applications to text | ||
RDRPOSTagger | 139 | over 4 years ago | A robust POS tagging toolkit available (in both Java & Python) together with pre-trained models for 40+ languages |
Lingua | 697 | 2 days ago | A language detection library for Kotlin and Java, suitable for long and short text alike |
Kotidgy | 3 | almost 6 years ago | — an index-based text data generator written in Kotlin |
Saul | 64 | over 4 years ago | Library for developing NLP systems, including built in modules like SRL, POS, etc |
ATR4S | 34 | about 6 years ago | Toolkit with state-of-the-art methods |
tm | 6 | about 8 years ago | Implementation of topic modeling based on regularized multilingual |
word2vec-scala | 11 | about 8 years ago | Scala interface to word2vec model; includes operations on vectors like word-distance and word-analogy |
Epic | 469 | over 4 years ago | Epic is a high performance statistical parser written in Scala, along with a framework for building complex structured prediction models |
Spark NLP | 3,827 | 9 days ago | Spark NLP is a natural language processing library built on top of Apache Spark ML that provides simple, performant & accurate NLP annotations for machine learning pipelines that scale easily in a distributed environment |
text2vec | 851 | about 2 months ago | Fast vectorization, topic modeling, distances and GloVe word embeddings in R |
wordVectors | 280 | over 2 years ago | An R package for creating and exploring word2vec and other word embedding models |
RMallet | 38 | about 2 years ago | R package to interface with the Java machine learning tool MALLET |
dfr-browser | 99 | over 2 years ago | Creates d3 visualizations for browsing topic models of text in a web browser |
dfrtopics | 47 | about 2 years ago | R package for exploring topic models of text |
sentiment_classifier | 172 | over 2 years ago | Sentiment Classification using Word Sense Disambiguation and WordNet Reader |
jProcessing | 148 | about 4 years ago | Japanese Natural Langauge Processing Libraries, with Japanese sentiment classification |
corporaexplorer | An R package for dynamic exploration of text collections | ||
tidytext | 1,182 | 6 months ago | Text mining using tidy tools |
spacyr | 250 | 5 months ago | R wrapper to spaCy NLP |
CRAN Task View: Natural Language Processing | 4 | about 1 year ago | |
Clojure-openNLP | 752 | almost 6 years ago | Natural Language Processing in Clojure (opennlp) |
Infections-clj | 217 | 5 months ago | Rails-like inflection library for Clojure and ClojureScript |
postagga | 159 | almost 4 years ago | A library to parse natural language in Clojure and ClojureScript |
A collection of Natural Language Processing (NLP) Ruby libraries, tools and software | 1,268 | over 1 year ago | Kevin Dias's |
Practical Natural Language Processing done in Ruby | 1,040 | over 1 year ago | |
whatlang | 968 | 7 months ago | — Natural language recognition library based on trigrams |
snips-nlu-rs | 340 | almost 2 years ago | A production ready library for intent parsing |
rust-bert | 2,593 | 13 days ago | Ready-to-use NLP pipelines and Transformer-based models |
VSCode Language Extension | NLP++ Language Extension for VSCode | ||
nlp-engine | 11 | 2 days ago | NLP++ engine to run NLP++ code on Linux including a full English parser |
VisualText | Homepage for the NLP++ Language | ||
NLP++ Wiki | Wiki entry for the NLP++ language | ||
CorpusLoaders | 32 | about 2 years ago | A variety of loaders for various NLP corpora |
Languages | 55 | 10 months ago | A package for working with human languages |
TextAnalysis | 374 | 19 days ago | Julia package for text analysis |
TextModels | 30 | 11 months ago | Neural Network based models for Natural Language Processing |
WordTokenizers | 97 | almost 3 years ago | High performance tokenizers for natural language processing and other related tasks |
Word2Vec | 62 | almost 3 years ago | Julia interface to word2vec |
awesome-nlp / Libraries / Services | |||
Wit-ai | 942 | about 3 years ago | Natural Language Interface for apps and devices |
IBM Watson's Natural Language Understanding | 135 | almost 2 years ago | API and Github demo |
Amazon Comprehend | NLP and ML suite covers most common tasks like NER, tagging, and sentiment analysis | ||
Google Cloud Natural Language API | Syntax Analysis, NER, Sentiment Analysis, and Content tagging in atleast 9 languages include English and Chinese (Simplified and Traditional) | ||
ParallelDots | High level Text Analysis API Service ranging from Sentiment Analysis to Intent Analysis | ||
Microsoft Cognitive Service | |||
TextRazor | |||
Rosette | |||
Textalytic | Natural Language Processing in the Browser with sentiment analysis, named entity extraction, POS tagging, word frequencies, topic modeling, word clouds, and more | ||
NLP Cloud | SpaCy NLP models (custom and pre-trained ones) served through a RESTful API for named entity recognition (NER), POS tagging, and more | ||
Cloudmersive | Unified and free NLP APIs that perform actions such as speech tagging, text rephrasing, language translation/detection, and sentence parsing | ||
awesome-nlp / Libraries / Annotation Tools | |||
GATE | General Architecture and Text Engineering is 15+ years old, free and open source | ||
Anafora | 240 | about 2 years ago | is free and open source, web-based raw text annotation tool |
brat | brat rapid annotation tool is an online environment for collaborative text annotation | ||
doccano | 9,466 | 24 days ago | doccano is free, open-source, and provides annotation features for text classification, sequence labeling and sequence to sequence |
INCEpTION | A semantic annotation platform offering intelligent assistance and knowledge management | ||
tagtog | , team-first web tool to find, create, maintain, and share datasets - costs $ | ||
prodigy | is an annotation tool powered by active learning, costs $ | ||
LightTag | Hosted and managed text annotation tool for teams, costs $ | ||
rstWeb | open source local or online tool for discourse tree annotations | ||
GitDox | open source server annotation tool with GitHub version control and validation for XML data and collaborative spreadsheet grids | ||
Label Studio | Hosted and managed text annotation tool for teams, freemium based, costs $ | ||
Datasaur | support various NLP tasks for individual or teams, freemium based | ||
Konfuzio | team-first hosted and on-prem text, image and PDF annotation tool powered by active learning, freemium based, costs $ | ||
UBIAI | Easy-to-use text annotation tool for teams with most comprehensive auto-annotation features. Supports NER, relations and document classification as well as OCR annotation for invoice labeling, costs $ | ||
Shoonya | 20 | 12 days ago | Shoonya is free and open source data annotation platform with wide varities of organization and workspace level management system. Shoonya is data agnostic, can be used by teams to annotate data with various level of verification stages at scale |
Annotation Lab | Free End-to-End No-Code platform for text annotation and DL model training/tuning. Out-of-the-box support for Named Entity Recognition, Classification, Relation extraction and Assertion Status Spark NLP models. Unlimited support for users, teams, projects, documents. Not FOSS | ||
FLAT | 110 | 3 months ago | FLAT is a web-based linguistic annotation environment based around the , a rich XML-based format for linguistic annotation. Free and open source |
awesome-nlp / Techniques / Text Embeddings | |||
Deep Contextualized Word Representations | ElMo - - - | ||
Universal Language Model Fine-tuning for Text Classification | ULMFiT - by Jeremy Howard and Sebastian Ruder | ||
Supervised Learning of Universal Sentence Representations from Natural Language Inference Data | InferSent - by facebook | ||
Learned in Translation: Contextualized Word Vectors | CoVe - | ||
Distributed Representations of Sentences and Documents | Pargraph vectors - from . See | ||
sense2vec | on word sense disambiguation | ||
Skip Thought Vectors | word representation method | ||
Adaptive skip-gram | similar approach, with adaptive properties | ||
Sequence to Sequence Learning | word vectors for machine translation | ||
awesome-nlp / Techniques / Question Answering and Knowledge Extraction | |||
DrQA | 4,475 | about 1 year ago | Open Domain Question Answering work by Facebook Research on Wikipedia data |
Document-QA | 433 | 8 months ago | Simple and Effective Multi-Paragraph Reading Comprehension by AllenAI |
Template-Based Information Extraction without the Templates | |||
Privee: An Architecture for Automatically Analyzing Web Privacy Policies | |||
awesome-nlp / Datasets | |||
nlp-datasets | 5,735 | over 1 year ago | great collection of nlp datasets |
gensim-data | 980 | over 6 years ago | Data repository for pretrained NLP models and NLP corpora |
awesome-nlp / Multilingual NLP Frameworks | |||
UDPipe | 361 | about 1 month ago | is a trainable pipeline for tokenizing, tagging, lemmatizing and parsing Universal Treebanks and other CoNLL-U files. Primarily written in C++, offers a fast and reliable solution for multilingual NLP processing |
NLP-Cube | 550 | 6 months ago | : Natural Language Processing Pipeline - Sentence Splitting, Tokenization, Lemmatization, Part-of-speech Tagging and Dependency Parsing. New platform, written in Python with Dynet 2.0. Offers standalone (CLI/Python bindings) and server functionality (REST API) |
UralicNLP | 70 | 2 months ago | is an NLP library mostly for many endangered Uralic languages such as Sami languages, Mordvin languages, Mari languages, Komi languages and so on. Also some non-endangered languages are supported such as Finnish together with non-Uralic languages such as Swedish and Arabic. UralicNLP can do morphological analysis, generation, lemmatization and disambiguation |
awesome-nlp / NLP in Korean / Libraries | |||
KoNLPy | Python package for Korean natural language processing | ||
Mecab (Korean) | C++ library for Korean NLP | ||
KoalaNLP | Scala library for Korean Natural Language Processing | ||
KoNLP | R package for Korean Natural language processing | ||
awesome-nlp / NLP in Korean / Blogs and Tutorials | |||
dsindex's blog | |||
Kangwon University's NLP course in Korean | |||
awesome-nlp / NLP in Korean / Datasets | |||
KAIST Corpus | A corpus from the Korea Advanced Institute of Science and Technology in Korean | ||
Naver Sentiment Movie Corpus in Korean | 560 | over 7 years ago | |
Chosun Ilbo archive | dataset in Korean from one of the major newspapers in South Korea, the Chosun Ilbo | ||
Chat data | 355 | over 1 year ago | Chatbot data in Korean |
Petitions | 40 | about 6 years ago | Collect expired petition data from the Blue House National Petition Site |
Korean Parallel corpora | 12 | almost 10 years ago | Neural Machine Translation(NMT) Dataset for & |
KorQuAD | Korean SQuAD dataset with Wiki HTML source. Mentions both v1.0 and v2.1 at the time of adding to Awesome NLP | ||
awesome-nlp / NLP in Arabic / Libraries | |||
goarabic | 107 | over 1 year ago | Go package for Arabic text processing |
jsastem | 26 | almost 12 years ago | Javascript for Arabic stemming |
PyArabic | Python libraries for Arabic | ||
RFTokenizer | 27 | 4 months ago | trainable Python segmenter for Arabic, Hebrew and Coptic |
awesome-nlp / NLP in Arabic / Datasets | |||
Multidomain Datasets | 114 | over 6 years ago | Largest Available Multi-Domain Resources for Arabic Sentiment Analysis |
LABR | 44 | almost 10 years ago | LArge Arabic Book Reviews dataset |
Arabic Stopwords | 305 | 7 months ago | A list of Arabic stopwords from various resources |
awesome-nlp / NLP in Chinese / Libraries | |||
jieba | 33,156 | about 2 months ago | Python package for Words Segmentation Utilities in Chinese |
SnowNLP | 6,403 | over 4 years ago | Python package for Chinese NLP |
FudanNLP | 2,645 | 11 months ago | Java library for Chinese text processing |
HanLP | 33,649 | 4 days ago | The multilingual NLP library |
awesome-nlp / NLP in Chinese / Anthology | |||
funNLP | 68,036 | 5 months ago | Collection of NLP tools and resources mainly for Chinese |
awesome-nlp / NLP in German | |||
German-NLP | 447 | about 2 months ago | Curated list of open-access/open-source/off-the-shelf resources and tools developed with a particular focus on German |
awesome-nlp / NLP in Polish | |||
Polish-NLP | 287 | about 3 years ago | A curated list of resources dedicated to Natural Language Processing (NLP) in polish. Models, tools, datasets |
awesome-nlp / NLP in Spanish / Libraries | |||
spanlp | 35 | 4 months ago | Python library to detect, censor and clean profanity, vulgarities, hateful words, racism, xenophobia and bullying in texts written in Spanish. It contains data of 21 Spanish-speaking countries |
awesome-nlp / NLP in Spanish / Data | |||
Columbian Political Speeches | 6 | about 11 years ago | |
Copenhagen Treebank | |||
Spanish Billion words corpus with Word2Vec embeddings | 45 | almost 2 years ago | |
Compilation of Spanish Unannotated Corpora | 92 | almost 2 years ago | |
awesome-nlp / NLP in Spanish / Word and Sentence Embeddings | |||
Spanish Word Embeddings Computed with Different Methods and from Different Corpora | 356 | about 5 years ago | |
Spanish Word Embeddings Computed from Large Corpora and Different Sizes Using fastText | 9 | about 5 years ago | |
Spanish Sentence Embeddings Computed from Large Corpora Using sent2vec | 4 | about 5 years ago | |
Beto - BERT for Spanish | 490 | 12 months ago | |
awesome-nlp / NLP in Indic languages / Data, Corpora and Treebanks | |||
Hindi Dependency Treebank | A multi-representational multi-layered treebank for Hindi and Urdu | ||
Universal Dependencies Treebank in Hindi | |||
awesome-nlp / NLP in Indic languages / Data, Corpora and Treebanks / Universal Dependencies Treebank in Hindi | |||
Parallel Universal Dependencies Treebank in Hindi | A smaller part of the above-mentioned treebank | ||
awesome-nlp / NLP in Indic languages / Data, Corpora and Treebanks | |||
ISI FIRE Stopwords List (Hindi and Bangla) | |||
Peter Graham's Stopwords List | 424 | over 1 year ago | |
NLTK Corpus | 60k Words POS Tagged, Bangla, Hindi, Marathi, Telugu | ||
Hindi Movie Reviews Dataset | 120 | about 4 years ago | ~1k Samples, 3 polarity classes |
BBC News Hindi Dataset | 218 | over 5 years ago | 4.3k Samples, 14 classes |
IIT Patna Hindi ABSA Dataset | 11 | about 7 years ago | 5.4k Samples, 12 Domains, 4k aspect terms, aspect and sentence level polarity in 4 classes |
Bangla ABSA | 0 | over 6 years ago | 5.5k Samples, 2 Domains, 10 aspect terms |
IIT Patna Movie Review Sentiment Dataset | 2k Samples, 3 polarity labels | ||
SAIL 2015 | Twitter and Facebook labelled sentiment samples in Hindi, Bengali, Tamil, Telugu | ||
IIT Bombay NLP Resources | Sentiwordnet, Movie and Tourism parallel labelled corpora, polarity labelled sense annotated corpus, Marathi polarity labelled corpus | ||
TDIL-IC aggregates a lot of useful resources and provides access to otherwise gated datasets | |||
awesome-nlp / NLP in Indic languages / Language Models and Word Embeddings | |||
Hindi2Vec | and ULMFIT style languge model | ||
IIT Patna Bilingual Word Embeddings Hi-En | |||
Fasttext word embeddings in a whole bunch of languages, trained on Common Crawl | |||
Hindi and Bengali Word2Vec | 2,217 | about 6 years ago | |
Hindi and Urdu Elmo Model | 1,458 | over 3 years ago | |
Sanskrit Albert | Trained on Sanskrit Wikipedia and OSCAR corpus | ||
awesome-nlp / NLP in Indic languages / Libraries and Tooling | |||
Multi-Task Deep Morphological Analyzer | 1 | almost 6 years ago | Deep Network based Morphological Parser for Hindi and Urdu |
Anoop Kunchukuttan | 550 | 4 months ago | 18 Languages, whole host of features from tokenization to translation |
SivaReddy's Dependency Parser | Dependency Parser and Pos Tagger for Kannada, Hindi and Telugu | ||
iNLTK | 814 | 9 months ago | A Natural Language Toolkit for Indic Languages (Indian subcontinent languages) built on top of Pytorch/Fastai, which aims to provide out of the box support for common NLP tasks |
awesome-nlp / NLP in Thai / Libraries | |||
PyThaiNLP | 976 | 3 days ago | Thai NLP in Python Package |
JTCC | 17 | over 7 years ago | A character cluster library in Java |
CutKum | 154 | 10 months ago | Word segmentation with deep learning in TensorFlow |
Thai Language Toolkit | Based on a paper by Wirote Aroonmanakun in 2002 with included dataset | ||
SynThai | 41 | over 7 years ago | Word segmentation and POS tagging using deep learning in Python |
awesome-nlp / NLP in Thai / Data | |||
Inter-BEST | A text corpus with 5 million words with word segmentation | ||
Prime Minister 29 | 139 | 8 months ago | Dataset containing speeches of the current Prime Minister of Thailand |
awesome-nlp / NLP in Danish | |||
Named Entity Recognition for Danish | 17 | about 5 years ago | |
DaNLP | 195 | 10 months ago | NLP resources in Danish |
Awesome Danish | 162 | 23 days ago | A curated list of awesome resources for Danish language technology |
awesome-nlp / NLP in Vietnamese / Libraries | |||
underthesea | 1,389 | 6 days ago | Vietnamese NLP Toolkit |
vn.vitk | 214 | over 2 years ago | A Vietnamese Text Processing Toolkit |
VnCoreNLP | 585 | over 1 year ago | A Vietnamese natural language processing toolkit |
PhoBERT | 654 | 3 months ago | Pre-trained language models for Vietnamese |
pyvi | 242 | 16 days ago | Python Vietnamese Core NLP Toolkit |
awesome-nlp / NLP in Vietnamese / Data | |||
Vietnamese treebank | 10,000 sentences for the constituency parsing task | ||
BKTreeBank | a Vietnamese Dependency Treebank | ||
UD_Vietnamese | 35 | 5 months ago | Vietnamese Universal Dependency Treebank |
VIVOS | a free Vietnamese speech corpus consisting of 15 hours of recording speech by AILab | ||
VNTQcorpus(big).txt | 1.75 million sentences in news | ||
ViText2SQL | 28 | 3 months ago | A dataset for Vietnamese Text-to-SQL semantic parsing (EMNLP-2020 Findings) |
EVB Corpus | 41 | over 5 years ago | 20,000,000 words (20 million) from 15 bilingual books, 100 parallel English-Vietnamese / Vietnamese-English texts, 250 parallel law and ordinance texts, 5,000 news articles, and 2,000 film subtitles |
awesome-nlp / NLP for Dutch | |||
python-frog | 47 | about 1 month ago | Python binding to Frog, an NLP suite for Dutch. (pos tagging, lemmatisation, dependency parsing, NER) |
SimpleNLG_NL | 5 | almost 4 years ago | Dutch surface realiser used for Natural Language Generation in Dutch, based on the SimpleNLG implementation for English and French |
Alpino | 22 | 19 days ago | Dependency parser for Dutch (also does PoS tagging and Lemmatisation) |
Kaldi NL | 65 | 12 months ago | Dutch Speech Recognition models based on |
spaCy | available. - Industrial strength NLP with Python and Cython | ||
awesome-nlp / NLP in Indonesian / Datasets | |||
ILPS | Kompas and Tempo collections at | ||
PANL10N for PoS tagging | : 39K sentences and 900K word tokens | ||
IDN for PoS tagging | 88 | over 2 years ago | : This corpus contains 10K sentences and 250K word tokens |
Indonesian Treebank | 36 | over 2 years ago | and |
IndoSum | 76 | over 5 years ago | for text summarization and classification both |
Wordnet-Bahasa | large, free, semantic dictionary | ||
IndoNLU | 538 | almost 2 years ago | IndoBenchmark includes pre-trained language model (IndoBERT), FastText model, Indo4B corpus, and several NLU benchmark datasets |
awesome-nlp / NLP in Indonesian / Libraries & Embedding | |||
bahasa | 19 | 5 months ago | Natural language toolkit |
Indonesian Word Embedding | 20 | over 6 years ago | |
Indonesian fastText Text Embedding | Pretrained trained on Wikipedia | ||
IndoNLU | 538 | almost 2 years ago | IndoBenchmark includes pretrained language model (IndoBERT), FastText model, Indo4B corpus, and several NLU benchmark datasets |
awesome-nlp / NLP in Urdu / Datasets | |||
Collection of Urdu datasets | 69 | 2 months ago | for POS, NER and NLP tasks |
awesome-nlp / NLP in Urdu / Libraries | |||
Natural Language Processing library | 281 | 9 months ago | for ( 🇵🇰)Urdu language |
awesome-nlp / NLP in Persian / Libraries | |||
Hazm | 1,191 | 3 months ago | Persian NLP Toolkit |
Parsivar | 229 | 9 months ago | : A Language Processing Toolkit for Persian |
Perke | 69 | 2 months ago | : Perke is a Python keyphrase extraction package for Persian language. It provides an end-to-end keyphrase extraction pipeline in which each component can be easily modified or extended to develop new models |
Perstem | 18 | over 8 years ago | : Persian stemmer, morphological analyzer, transliterator, and partial part-of-speech tagger |
ParsiAnalyzer | 151 | 6 months ago | : Persian Analyzer For Elasticsearch |
virastar | 82 | over 6 years ago | : Cleaning up Persian text! |
awesome-nlp / NLP in Persian / Datasets | |||
Bijankhan Corpus | : Bijankhan corpus is a tagged corpus that is suitable for natural language processing research on the Persian (Farsi) language. This collection is gathered form daily news and common texts. In this collection all documents are categorized into different subjects such as political, cultural and so on. Totally, there are 4300 different subjects. The Bijankhan collection contains about 2.6 millions manually tagged words with a tag set that contains 40 Persian POS tags | ||
Uppsala Persian Corpus (UPC) | : Uppsala Persian Corpus (UPC) is a large, freely available Persian corpus. The corpus is a modified version of the Bijankhan corpus with additional sentence segmentation and consistent tokenization containing 2,704,028 tokens and annotated with 31 part-of-speech tags. The part-of-speech tags are listed with explanations in | ||
Large-Scale Colloquial Persian | : Large Scale Colloquial Persian Dataset (LSCP) is hierarchically organized in asemantic taxonomy that focuses on multi-task informal Persian language understanding as a comprehensive problem. LSCP includes 120M sentences from 27M casual Persian tweets with its dependency relations in syntactic annotation, Part-of-speech tags, sentiment polarity and automatic translation of original Persian sentences in English (EN), German (DE), Czech (CS), Italian (IT) and Hindi (HI) spoken languages. Learn more about this project at | ||
ArmanPersoNERCorpus | 56 | about 4 years ago | : The dataset includes 250,015 tokens and 7,682 Persian sentences in total. It is available in 3 folds to be used in turn as training and test sets. Each file contains one token, along with its manually annotated named-entity tag, per line. Each sentence is separated with a newline. The NER tags are in IOB format |
FarsiYar PersianNER | 224 | over 3 years ago | : The dataset includes about 25,000,000 tokens and about 1,000,000 Persian sentences in total based on . The NER tags are in IOB format. More than 1000 volunteers contributed tag improvements to this dataset via web panel or android app. They release updated tags every two weeks |
PERLEX | : The first Persian dataset for relation extraction, which is an expert translated version of the “Semeval-2010-Task-8” dataset. Link to the relevant publication | ||
Persian Syntactic Dependency Treebank | : This treebank is supplied for free noncommercial use. For commercial uses feel free to contact us. The number of annotated sentences is 29,982 sentences including samples from almost all verbs of the Persian valency lexicon | ||
Uppsala Persian Dependency Treebank (UPDT) | : Dependency-based syntactically annotated corpus | ||
Hamshahri | : Hamshahri collection is a standard reliable Persian text collection that was used at Cross Language Evaluation Forum (CLEF) during years 2008 and 2009 for evaluation of Persian information retrieval systems | ||
awesome-nlp / NLP in Ukrainian | |||
awesome-ukrainian-nlp | 160 | 7 months ago | a curated list of Ukrainian NLP datasets, models, etc |
UkrainianLT | 30 | over 2 years ago | another curated list with a focus on machine translation and speech processing |
awesome-nlp / NLP in Hungarian | |||
awesome-hungarian-nlp | 222 | 12 months ago | : A curated list of free resources dedicated to Hungarian Natural Language Processing |
awesome-nlp / NLP in Portuguese | |||
Portuguese-nlp | 227 | 3 months ago | a List of resources and tools developed with focus on Portuguese |
awesome-nlp / Other Languages | |||
pymorphy2 | 1,117 | 4 months ago | Russian: - a good pos-tagger for Russian |
ICU Tokenizer | Asian Languages: Thai, Lao, Chinese, Japanese, and Korean implementation in ElasticSearch | ||
CLTK | 835 | about 2 months ago | Ancient Languages: : The Classical Language Toolkit is a Python library and collection of texts for doing NLP in ancient languages |
NLPH_Resources | 246 | 2 months ago | Hebrew: - A collection of papers, corpora and linguistic resources for NLP in Hebrew |