awesome-nlp

NLP resource repository

A curated collection of resources and references for Natural Language Processing

book A curated list of resources dedicated to Natural Language Processing (NLP)

GitHub
17k stars
610 watching
3k forks
last commit: about 1 year ago
Linked from 8 awesome lists

awesomeawesome-listdeep-learninglanguagemachine-learningnatural-language-processingnlptext-mining

NLP-Overview is an up-to-date overview of deep learning techniques applied to NLP, including theory, implementations, applications, and state-of-the-art results. This is a great Deep NLP Introduction for researchers
NLP-Progress tracks the progress in Natural Language Processing, including the datasets and the current state-of-the-art for the most common NLP tasks
NLP's ImageNet moment has arrived
ACL 2018 Highlights: Understanding Representation and Evaluation in More Challenging Settings
Four deep learning trends from ACL 2017. Part One: Linguistic Structure and Word Embeddings
Four deep learning trends from ACL 2017. Part Two: Interpretability and Attention
Highlights of EMNLP 2017: Exciting Datasets, Return of the Clusters, and More!
Deep Learning for Natural Language Processing (NLP): Advancements & Trends
Survey of the State of the Art in Natural Language Generation

awesome-nlp / Prominent NLP Research Labs
The Berkeley NLP Group Notable contributions include a tool to reconstruct long dead languages, referenced and by taking corpora from 637 languages currently spoken in Asia and the Pacific and recreating their descendant
Language Technologies Institute, Carnegie Mellon University Notable projects include , a syntax driven machine translation system for endangered languages like Quechua and Aymara and previously, which created to improve NLP tools for Arabic
NLP research group, Columbia University Responsible for creating BOLT ( interactive error handling for speech translation systems) and an un-named project to characterize laughter in dialogue
The Center or Language and Speech Processing, John Hopkins University Recently in the news for developing speech recognition software to create a diagnostic test or Parkinson's Disease,
Computational Linguistics and Information Processing Group, University of Maryland Notable contributions include and modeling development of phonetic representations
Penn Natural Language Processing, University of Pennsylvania Famous for creating the
The Stanford Nautral Language Processing Group One of the top NLP research labs in the world, notable for creating and their

awesome-nlp / Tutorials / Reading Content
Machine Learning 101 from Google's Senior Creative Engineer explains Machine Learning for engineer's and executives alike
AI Playbook a16z AI playbook is a great link to forward to your managers or content for your presentations
Ruder's Blog by for commentary on the best of NLP Research
How To Label Data guide to managing larger linguistic annotation projects
Depends on the Definition collection of blog posts covering a wide array of NLP topics with detailed implementation
Understand & Implement Natural Language Processing
NLP in Python Collection of Github notebooks
Natural Language Processing: An Introduction Oxford
Deep Learning for NLP with Pytorch
Hands-On NLTK Tutorial 545 6 months ago NLTK Tutorials, Jupyter notebooks
Natural Language Processing with Python – Analyzing Text with the Natural Language Toolkit An online and print book introducing NLP concepts using NLTK. The book's authors also wrote the NLTK library
Train a new language model from scratch Hugging Face 🤗
The Super Duper NLP Repo (SDNLPR) : Collection of Colab notebooks covering a wide array of NLP task implementations
Deep Learning, NLP, and Representations
The Illustrated BERT, ELMo, and co. (How NLP Cracked Transfer Learning) and
Natural Language Processing by Hal Daumé III
arXiv: Natural Language Processing (Almost) from Scratch
Karpathy's The Unreasonable Effectiveness of Recurrent Neural Networks
Machine Learning Mastery: Deep Learning for Natural Language Processing
Visual NLP Paper Summaries

awesome-nlp / Tutorials / Videos and Online Courses
Advanced Natural Language Processing CS 685, UMass Amherst CS
Deep Natural Language Processing 15,683 over 1 year ago Lectures series from Oxford
Deep Learning for Natural Language Processing (cs224-n) Richard Socher and Christopher Manning's Stanford Course
Neural Networks for NLP Carnegie Mellon Language Technology Institute there
Deep NLP Course 9,811 7 days ago by Yandex Data School, covering important ideas from text embedding to machine translation including sequence modeling, language models and so on
fast.ai Code-First Intro to Natural Language Processing This covers a blend of traditional NLP topics (including regex, SVD, naive bayes, tokenization) and recent neural network approaches (including RNNs, seq2seq, GRUs, and the Transformer), as well as addressing urgent ethical issues, such as bias and disinformation. Find the Jupyter Notebooks
Machine Learning University - Accelerated Natural Language Processing Lectures go from introduction to NLP and text processing to Recurrent Neural Networks and Transformers. Material can be found
Applied Natural Language Processing Lecture series from IIT Madras taking from the basics all the way to autoencoders and everything. The github notebooks for this course are also available

awesome-nlp / Tutorials / Books
Speech and Language Processing free, by Prof. Dan Jurafsy
Natural Language Processing 4,946 almost 2 years ago free, NLP notes by Dr. Jacob Eisenstein at GeorgiaTech
NLP with PyTorch 1,991 over 1 year ago Brian & Delip Rao
Text Mining in R
Natural Language Processing with Python
Practical Natural Language Processing
Natural Language Processing with Spark NLP
Deep Learning for Natural Language Processing by Stephan Raaijmakers
Real-World Natural Language Processing by Masato Hagiwara
Natural Language Processing in Action, Second Edition by Hobson Lane and Maria Dyshel

awesome-nlp / Libraries
Twitter-text 3,081 7 months ago A JavaScript implementation of Twitter's text processing library
Knwl.js 5,292 about 1 year ago A Natural Language Processor in JS
Retext 2,360 about 1 month ago Extensible system for analyzing and manipulating natural language
NLP Compromise 11,482 about 1 month ago Natural Language processing in the browser
Natural 10,625 3 months ago general natural language facilities for node
Poplar 519 almost 2 years ago A web-based annotation tool for natural language processing (NLP)
NLP.js 6,283 5 months ago An NLP library for building bots
node-question-answering 466 over 1 year ago Fast and production-ready question answering w/ DistilBERT in Node.js
sentimental-onix 3 21 days ago Sentiment models for spacy using onnx
TextAttack 2,973 4 months ago Adversarial attacks, adversarial training, and data augmentation in NLP
TextBlob Providing a consistent API for diving into common natural language processing (NLP) tasks. Stands on the giant shoulders of and , and plays nicely with both
spaCy 30,230 29 days ago Industrial strength NLP with Python and Cython
Speedster 8,375 4 months ago Automatically apply SOTA optimization techniques to achieve the maximum inference speed-up on your hardware

awesome-nlp / Libraries / Speedster
textacy 2,217 about 1 year ago Higher level NLP built on spaCy

awesome-nlp / Libraries
gensim Python library to conduct unsupervised semantic modelling from plain text
scattertext 2,249 about 2 months ago Python library to produce d3 visualizations of how language differs between corpora
GluonNLP 2,557 about 1 year ago A deep learning toolkit for NLP, built on MXNet/Gluon, for research prototyping and industrial deployment of state-of-the-art models on a wide range of NLP tasks
AllenNLP 11,757 almost 2 years ago An NLP research library, built on PyTorch, for developing state-of-the-art deep learning models on a wide variety of linguistic tasks
PyTorch-NLP 2,213 over 1 year ago NLP research toolkit designed to support rapid prototyping with better data loaders, word vector loaders, neural network layer representations, common NLP metrics such as BLEU
Rosetta 206 about 2 years ago Text processing tools and wrappers (e.g. Vowpal Wabbit)
PyNLPl 479 about 1 year ago Python Natural Language Processing Library. General purpose NLP library for Python, handles some specific formats like ARPA language models, Moses phrasetables, GIZA++ alignments
foliapy 18 about 1 month ago Python library for working with , an XML format for linguistic annotation
PySS3 336 over 1 year ago Python package that implements a novel white-box machine learning model for text classification, called SS3. Since SS3 has the ability to visually explain its rationale, this package also comes with easy-to-use interactive visualizations tools ( )
jPTDP 158 over 5 years ago A toolkit for joint part-of-speech (POS) tagging and dependency parsing. jPTDP provides pre-trained models for 40+ languages
BigARTM 662 over 1 year ago a fast library for topic modelling
Snips NLU 3,897 over 1 year ago A production ready library for intent parsing
Chazutsu 243 about 2 years ago A library for downloading&parsing standard NLP research datasets
Word Forms 623 over 3 years ago Word forms can accurately generate all possible forms of an English word
Multilingual Latent Dirichlet Allocation (LDA) 82 4 months ago A multilingual and extensible document clustering pipeline
Natural Language Toolkit (NLTK) A library containing a wide variety of NLP functionality, supporting over 50 corpora
NLP Architect 2,942 about 2 years ago A library for exploring the state-of-the-art deep learning topologies and techniques for NLP and NLU
Flair 13,939 6 days ago A very simple framework for state-of-the-art multilingual NLP built on PyTorch. Includes BERT, ELMo and Flair embeddings
Kashgari 2,395 3 months ago Simple, Keras-powered multilingual NLP framework, allows you to build your models in 5 minutes for named entity recognition (NER), part-of-speech tagging (PoS) and text classification tasks. Includes BERT and word2vec embedding
FARM 1,741 11 months ago Fast & easy transfer learning for NLP. Harvesting language models for the industry. Focus on Question Answering
Haystack 17,691 6 days ago End-to-end Python framework for building natural language search interfaces to data. Leverages Transformers and the State-of-the-Art of NLP. Supports DPR, Elasticsearch, HuggingFace’s Modelhub, and much more!
Rita DSL 65 about 2 years ago a DSL, loosely based on . Allows to define language patterns (rule-based NLP) which are then translated into , or if you prefer less features and lightweight - regex patterns
Transformers 135,022 5 days ago Natural Language Processing for TensorFlow 2.0 and PyTorch
Tokenizers 9,051 6 days ago Tokenizers optimized for Research and Production
fairSeq 30,522 about 1 month ago Facebook AI Research implementations of SOTA seq2seq models in Pytorch
corex_topic 627 over 3 years ago Hierarchical Topic Modeling with Minimal Domain Knowledge
Sockeye 1,212 28 days ago Neural Machine Translation (NMT) toolkit that powers Amazon Translate
DL Translate 453 3 months ago A deep learning-based translation library for 50 languages, built on and Facebook's mBART Large
Jury 188 3 months ago Evaluation of NLP model outputs offering various automated metrics
python-ucto 29 2 months ago Unicode-aware regular-expression based tokenizer for various languages. Python binding to C++ library, supports
InsNet 66 about 3 years ago A neural network library for building instance-dependent NLP models with padding-free dynamic batching
MIT Information Extraction Toolkit 2,921 about 2 years ago C, C++, and Python tools for named entity recognition and relation extraction
CRF++ Open source implementation of Conditional Random Fields (CRFs) for segmenting/labeling sequential data & other Natural Language Processing tasks
CRFsuite CRFsuite is an implementation of Conditional Random Fields (CRFs) for labeling sequential data
BLLIP Parser 227 about 3 years ago BLLIP Natural Language Parser (also known as the Charniak-Johnson parser)
colibri-core 124 about 1 year ago C++ library, command line tools, and Python binding for extracting and working with basic linguistic constructions such as n-grams and skipgrams in a quick and memory-efficient way
ucto 65 4 days ago Unicode-aware regular-expression based tokenizer for various languages. Tool and C++ library. Supports FoLiA format
libfolia 15 4 days ago C++ library for the
frog 75 4 days ago Memory-based NLP suite developed for Dutch: PoS tagger, lemmatiser, dependency parser, NER, shallow parser, morphological analyzer
MeTA 696 over 1 year ago is a C++ Data Sciences Toolkit that facilitates mining big text data
Mecab (Japanese)
Moses
StarSpace 3,946 almost 2 years ago a library from Facebook for creating embeddings of word-level, paragraph-level, document-level and for text classification
Stanford NLP
OpenNLP
NLP4J
Word2vec in Java
ReVerb 543 over 5 years ago Web-Scale Open Information Extraction
OpenRegex 75 over 10 years ago An efficient and flexible token-based regular expression language and engine
CogcompNLP 472 over 1 year ago Core libraries developed in the U of Illinois' Cognitive Computation Group
MALLET MAchine Learning for LanguagE Toolkit - package for statistical natural language processing, document classification, clustering, topic modeling, information extraction, and other machine learning applications to text
RDRPOSTagger 140 almost 5 years ago A robust POS tagging toolkit available (in both Java & Python) together with pre-trained models for 40+ languages
Lingua 707 6 days ago A language detection library for Kotlin and Java, suitable for long and short text alike
Kotidgy 3 almost 6 years ago — an index-based text data generator written in Kotlin
Saul 64 almost 5 years ago Library for developing NLP systems, including built in modules like SRL, POS, etc
ATR4S 35 over 6 years ago Toolkit with state-of-the-art methods
tm 7 about 8 years ago Implementation of topic modeling based on regularized multilingual
word2vec-scala 11 over 8 years ago Scala interface to word2vec model; includes operations on vectors like word-distance and word-analogy
Epic 469 almost 5 years ago Epic is a high performance statistical parser written in Scala, along with a framework for building complex structured prediction models
Spark NLP 3,871 6 days ago Spark NLP is a natural language processing library built on top of Apache Spark ML that provides simple, performant & accurate NLP annotations for machine learning pipelines that scale easily in a distributed environment
text2vec 853 3 months ago Fast vectorization, topic modeling, distances and GloVe word embeddings in R
wordVectors 282 over 2 years ago An R package for creating and exploring word2vec and other word embedding models
RMallet 38 over 2 years ago R package to interface with the Java machine learning tool MALLET
dfr-browser 99 over 2 years ago Creates d3 visualizations for browsing topic models of text in a web browser
dfrtopics 47 over 2 years ago R package for exploring topic models of text
sentiment_classifier 172 over 2 years ago Sentiment Classification using Word Sense Disambiguation and WordNet Reader
jProcessing 148 about 4 years ago Japanese Natural Langauge Processing Libraries, with Japanese sentiment classification
corporaexplorer An R package for dynamic exploration of text collections
tidytext 1,180 7 months ago Text mining using tidy tools
spacyr 251 6 months ago R wrapper to spaCy NLP
CRAN Task View: Natural Language Processing 4 about 1 year ago
Clojure-openNLP 753 almost 6 years ago Natural Language Processing in Clojure (opennlp)
Infections-clj 217 7 months ago Rails-like inflection library for Clojure and ClojureScript
postagga 159 almost 4 years ago A library to parse natural language in Clojure and ClojureScript
A collection of Natural Language Processing (NLP) Ruby libraries, tools and software 1,269 over 1 year ago Kevin Dias's
Practical Natural Language Processing done in Ruby 1,042 over 1 year ago
whatlang 970 8 months ago — Natural language recognition library based on trigrams
snips-nlu-rs 340 almost 2 years ago A production ready library for intent parsing
rust-bert 2,651 about 2 months ago Ready-to-use NLP pipelines and Transformer-based models
VSCode Language Extension NLP++ Language Extension for VSCode
nlp-engine 12 24 days ago NLP++ engine to run NLP++ code on Linux including a full English parser
VisualText Homepage for the NLP++ Language
NLP++ Wiki Wiki entry for the NLP++ language
CorpusLoaders 32 about 2 years ago A variety of loaders for various NLP corpora
Languages 54 12 months ago A package for working with human languages
TextAnalysis 373 about 2 months ago Julia package for text analysis
TextModels 29 about 1 year ago Neural Network based models for Natural Language Processing
WordTokenizers 96 almost 3 years ago High performance tokenizers for natural language processing and other related tasks
Word2Vec 61 about 3 years ago Julia interface to word2vec

awesome-nlp / Libraries / Services
Wit-ai 942 about 3 years ago Natural Language Interface for apps and devices
IBM Watson's Natural Language Understanding 135 almost 2 years ago API and Github demo
Amazon Comprehend NLP and ML suite covers most common tasks like NER, tagging, and sentiment analysis
Google Cloud Natural Language API Syntax Analysis, NER, Sentiment Analysis, and Content tagging in atleast 9 languages include English and Chinese (Simplified and Traditional)
ParallelDots High level Text Analysis API Service ranging from Sentiment Analysis to Intent Analysis
Microsoft Cognitive Service
TextRazor
Rosette
Textalytic Natural Language Processing in the Browser with sentiment analysis, named entity extraction, POS tagging, word frequencies, topic modeling, word clouds, and more
NLP Cloud SpaCy NLP models (custom and pre-trained ones) served through a RESTful API for named entity recognition (NER), POS tagging, and more
Cloudmersive Unified and free NLP APIs that perform actions such as speech tagging, text rephrasing, language translation/detection, and sentence parsing

awesome-nlp / Libraries / Annotation Tools
GATE General Architecture and Text Engineering is 15+ years old, free and open source
Anafora 241 about 2 years ago is free and open source, web-based raw text annotation tool
brat brat rapid annotation tool is an online environment for collaborative text annotation
doccano 9,572 23 days ago doccano is free, open-source, and provides annotation features for text classification, sequence labeling and sequence to sequence
INCEpTION A semantic annotation platform offering intelligent assistance and knowledge management
tagtog , team-first web tool to find, create, maintain, and share datasets - costs $
prodigy is an annotation tool powered by active learning, costs $
LightTag Hosted and managed text annotation tool for teams, costs $
rstWeb open source local or online tool for discourse tree annotations
GitDox open source server annotation tool with GitHub version control and validation for XML data and collaborative spreadsheet grids
Label Studio Hosted and managed text annotation tool for teams, freemium based, costs $
Datasaur support various NLP tasks for individual or teams, freemium based
Konfuzio team-first hosted and on-prem text, image and PDF annotation tool powered by active learning, freemium based, costs $
UBIAI Easy-to-use text annotation tool for teams with most comprehensive auto-annotation features. Supports NER, relations and document classification as well as OCR annotation for invoice labeling, costs $
Shoonya 21 8 days ago Shoonya is free and open source data annotation platform with wide varities of organization and workspace level management system. Shoonya is data agnostic, can be used by teams to annotate data with various level of verification stages at scale
Annotation Lab Free End-to-End No-Code platform for text annotation and DL model training/tuning. Out-of-the-box support for Named Entity Recognition, Classification, Relation extraction and Assertion Status Spark NLP models. Unlimited support for users, teams, projects, documents. Not FOSS
FLAT 110 5 months ago FLAT is a web-based linguistic annotation environment based around the , a rich XML-based format for linguistic annotation. Free and open source

awesome-nlp / Techniques / Text Embeddings
Deep Contextualized Word Representations ElMo - - -
Universal Language Model Fine-tuning for Text Classification ULMFiT - by Jeremy Howard and Sebastian Ruder
Supervised Learning of Universal Sentence Representations from Natural Language Inference Data InferSent - by facebook
Learned in Translation: Contextualized Word Vectors CoVe -
Distributed Representations of Sentences and Documents Pargraph vectors - from . See
sense2vec on word sense disambiguation
Skip Thought Vectors word representation method
Adaptive skip-gram similar approach, with adaptive properties
Sequence to Sequence Learning word vectors for machine translation

awesome-nlp / Techniques / Question Answering and Knowledge Extraction
DrQA 4,478 about 1 year ago Open Domain Question Answering work by Facebook Research on Wikipedia data
Document-QA 434 10 months ago Simple and Effective Multi-Paragraph Reading Comprehension by AllenAI
Template-Based Information Extraction without the Templates
Privee: An Architecture for Automatically Analyzing Web Privacy Policies

awesome-nlp / Datasets
nlp-datasets 5,775 almost 2 years ago great collection of nlp datasets
gensim-data 988 over 6 years ago Data repository for pretrained NLP models and NLP corpora

awesome-nlp / Multilingual NLP Frameworks
UDPipe 364 8 days ago is a trainable pipeline for tokenizing, tagging, lemmatizing and parsing Universal Treebanks and other CoNLL-U files. Primarily written in C++, offers a fast and reliable solution for multilingual NLP processing
NLP-Cube 554 18 days ago : Natural Language Processing Pipeline - Sentence Splitting, Tokenization, Lemmatization, Part-of-speech Tagging and Dependency Parsing. New platform, written in Python with Dynet 2.0. Offers standalone (CLI/Python bindings) and server functionality (REST API)
UralicNLP 70 16 days ago is an NLP library mostly for many endangered Uralic languages such as Sami languages, Mordvin languages, Mari languages, Komi languages and so on. Also some non-endangered languages are supported such as Finnish together with non-Uralic languages such as Swedish and Arabic. UralicNLP can do morphological analysis, generation, lemmatization and disambiguation

awesome-nlp / NLP in Korean / Libraries
KoNLPy Python package for Korean natural language processing
Mecab (Korean) C++ library for Korean NLP
KoalaNLP Scala library for Korean Natural Language Processing
KoNLP R package for Korean Natural language processing

awesome-nlp / NLP in Korean / Blogs and Tutorials
dsindex's blog
Kangwon University's NLP course in Korean

awesome-nlp / NLP in Korean / Datasets
KAIST Corpus A corpus from the Korea Advanced Institute of Science and Technology in Korean
Naver Sentiment Movie Corpus in Korean 566 over 7 years ago
Chosun Ilbo archive dataset in Korean from one of the major newspapers in South Korea, the Chosun Ilbo
Chat data 355 over 1 year ago Chatbot data in Korean
Petitions 40 over 6 years ago Collect expired petition data from the Blue House National Petition Site
Korean Parallel corpora 12 almost 10 years ago Neural Machine Translation(NMT) Dataset for &
KorQuAD Korean SQuAD dataset with Wiki HTML source. Mentions both v1.0 and v2.1 at the time of adding to Awesome NLP

awesome-nlp / NLP in Arabic / Libraries
goarabic 108 over 1 year ago Go package for Arabic text processing
jsastem 26 almost 12 years ago Javascript for Arabic stemming
PyArabic Python libraries for Arabic
RFTokenizer 27 5 months ago trainable Python segmenter for Arabic, Hebrew and Coptic

awesome-nlp / NLP in Arabic / Datasets
Multidomain Datasets 114 over 6 years ago Largest Available Multi-Domain Resources for Arabic Sentiment Analysis
LABR 44 about 10 years ago LArge Arabic Book Reviews dataset
Arabic Stopwords 308 8 months ago A list of Arabic stopwords from various resources

awesome-nlp / NLP in Chinese / Libraries
jieba 33,351 3 months ago Python package for Words Segmentation Utilities in Chinese
SnowNLP 6,436 almost 5 years ago Python package for Chinese NLP
FudanNLP 2,647 about 1 year ago Java library for Chinese text processing
HanLP 33,920 4 days ago The multilingual NLP library

awesome-nlp / NLP in Chinese / Anthology
funNLP 69,165 7 months ago Collection of NLP tools and resources mainly for Chinese

awesome-nlp / NLP in German
German-NLP 451 22 days ago Curated list of open-access/open-source/off-the-shelf resources and tools developed with a particular focus on German

awesome-nlp / NLP in Polish
Polish-NLP 294 over 3 years ago A curated list of resources dedicated to Natural Language Processing (NLP) in polish. Models, tools, datasets

awesome-nlp / NLP in Spanish / Libraries
spanlp 36 5 months ago Python library to detect, censor and clean profanity, vulgarities, hateful words, racism, xenophobia and bullying in texts written in Spanish. It contains data of 21 Spanish-speaking countries

awesome-nlp / NLP in Spanish / Data
Columbian Political Speeches 6 over 11 years ago
Copenhagen Treebank
Spanish Billion words corpus with Word2Vec embeddings 45 almost 2 years ago
Compilation of Spanish Unannotated Corpora 92 about 2 years ago

awesome-nlp / NLP in Spanish / Word and Sentence Embeddings
Spanish Word Embeddings Computed with Different Methods and from Different Corpora 356 about 5 years ago
Spanish Word Embeddings Computed from Large Corpora and Different Sizes Using fastText 9 over 5 years ago
Spanish Sentence Embeddings Computed from Large Corpora Using sent2vec 4 over 5 years ago
Beto - BERT for Spanish 492 about 1 year ago

awesome-nlp / NLP in Indic languages / Data, Corpora and Treebanks
Hindi Dependency Treebank A multi-representational multi-layered treebank for Hindi and Urdu
Universal Dependencies Treebank in Hindi

awesome-nlp / NLP in Indic languages / Data, Corpora and Treebanks / Universal Dependencies Treebank in Hindi
Parallel Universal Dependencies Treebank in Hindi A smaller part of the above-mentioned treebank

awesome-nlp / NLP in Indic languages / Data, Corpora and Treebanks
ISI FIRE Stopwords List (Hindi and Bangla)
Peter Graham's Stopwords List 424 over 1 year ago
NLTK Corpus 60k Words POS Tagged, Bangla, Hindi, Marathi, Telugu
Hindi Movie Reviews Dataset 122 over 4 years ago ~1k Samples, 3 polarity classes
BBC News Hindi Dataset 218 over 5 years ago 4.3k Samples, 14 classes
IIT Patna Hindi ABSA Dataset 11 about 7 years ago 5.4k Samples, 12 Domains, 4k aspect terms, aspect and sentence level polarity in 4 classes
Bangla ABSA 0 over 6 years ago 5.5k Samples, 2 Domains, 10 aspect terms
IIT Patna Movie Review Sentiment Dataset 2k Samples, 3 polarity labels
SAIL 2015 Twitter and Facebook labelled sentiment samples in Hindi, Bengali, Tamil, Telugu
IIT Bombay NLP Resources Sentiwordnet, Movie and Tourism parallel labelled corpora, polarity labelled sense annotated corpus, Marathi polarity labelled corpus
TDIL-IC aggregates a lot of useful resources and provides access to otherwise gated datasets

awesome-nlp / NLP in Indic languages / Language Models and Word Embeddings
Hindi2Vec and ULMFIT style languge model
IIT Patna Bilingual Word Embeddings Hi-En
Fasttext word embeddings in a whole bunch of languages, trained on Common Crawl
Hindi and Bengali Word2Vec 2,215 about 6 years ago
Hindi and Urdu Elmo Model 1,463 over 3 years ago
Sanskrit Albert Trained on Sanskrit Wikipedia and OSCAR corpus

awesome-nlp / NLP in Indic languages / Libraries and Tooling
Multi-Task Deep Morphological Analyzer 1 about 6 years ago Deep Network based Morphological Parser for Hindi and Urdu
Anoop Kunchukuttan 556 6 months ago 18 Languages, whole host of features from tokenization to translation
SivaReddy's Dependency Parser Dependency Parser and Pos Tagger for Kannada, Hindi and Telugu
iNLTK 822 10 months ago A Natural Language Toolkit for Indic Languages (Indian subcontinent languages) built on top of Pytorch/Fastai, which aims to provide out of the box support for common NLP tasks

awesome-nlp / NLP in Thai / Libraries
PyThaiNLP 987 6 days ago Thai NLP in Python Package
JTCC 18 over 7 years ago A character cluster library in Java
CutKum 154 11 months ago Word segmentation with deep learning in TensorFlow
Thai Language Toolkit Based on a paper by Wirote Aroonmanakun in 2002 with included dataset
SynThai 41 over 7 years ago Word segmentation and POS tagging using deep learning in Python

awesome-nlp / NLP in Thai / Data
Inter-BEST A text corpus with 5 million words with word segmentation
Prime Minister 29 141 9 months ago Dataset containing speeches of the current Prime Minister of Thailand

awesome-nlp / NLP in Danish
Named Entity Recognition for Danish 17 over 5 years ago
DaNLP 198 11 months ago NLP resources in Danish
Awesome Danish 165 17 days ago A curated list of awesome resources for Danish language technology

awesome-nlp / NLP in Vietnamese / Libraries
underthesea 1,414 25 days ago Vietnamese NLP Toolkit
vn.vitk 214 almost 3 years ago A Vietnamese Text Processing Toolkit
VnCoreNLP 592 almost 2 years ago A Vietnamese natural language processing toolkit
PhoBERT 663 4 months ago Pre-trained language models for Vietnamese
pyvi 245 about 2 months ago Python Vietnamese Core NLP Toolkit

awesome-nlp / NLP in Vietnamese / Data
Vietnamese treebank 10,000 sentences for the constituency parsing task
BKTreeBank a Vietnamese Dependency Treebank
UD_Vietnamese 36 9 days ago Vietnamese Universal Dependency Treebank
VIVOS a free Vietnamese speech corpus consisting of 15 hours of recording speech by AILab
VNTQcorpus(big).txt 1.75 million sentences in news
ViText2SQL 28 4 months ago A dataset for Vietnamese Text-to-SQL semantic parsing (EMNLP-2020 Findings)
EVB Corpus 42 over 5 years ago 20,000,000 words (20 million) from 15 bilingual books, 100 parallel English-Vietnamese / Vietnamese-English texts, 250 parallel law and ordinance texts, 5,000 news articles, and 2,000 film subtitles

awesome-nlp / NLP for Dutch
python-frog 47 24 days ago Python binding to Frog, an NLP suite for Dutch. (pos tagging, lemmatisation, dependency parsing, NER)
SimpleNLG_NL 5 about 4 years ago Dutch surface realiser used for Natural Language Generation in Dutch, based on the SimpleNLG implementation for English and French
Alpino 22 7 days ago Dependency parser for Dutch (also does PoS tagging and Lemmatisation)
Kaldi NL 66 about 1 year ago Dutch Speech Recognition models based on
spaCy available. - Industrial strength NLP with Python and Cython

awesome-nlp / NLP in Indonesian / Datasets
ILPS Kompas and Tempo collections at
PANL10N for PoS tagging : 39K sentences and 900K word tokens
IDN for PoS tagging 88 over 2 years ago : This corpus contains 10K sentences and 250K word tokens
Indonesian Treebank 36 over 2 years ago and
IndoSum 76 over 5 years ago for text summarization and classification both
Wordnet-Bahasa large, free, semantic dictionary
IndoNLU 556 5 days ago IndoBenchmark includes pre-trained language model (IndoBERT), FastText model, Indo4B corpus, and several NLU benchmark datasets

awesome-nlp / NLP in Indonesian / Libraries & Embedding
bahasa 19 6 months ago Natural language toolkit
Indonesian Word Embedding 20 almost 7 years ago
Indonesian fastText Text Embedding Pretrained trained on Wikipedia
IndoNLU 556 5 days ago IndoBenchmark includes pretrained language model (IndoBERT), FastText model, Indo4B corpus, and several NLU benchmark datasets

awesome-nlp / NLP in Urdu / Datasets
Collection of Urdu datasets 71 3 months ago for POS, NER and NLP tasks

awesome-nlp / NLP in Urdu / Libraries
Natural Language Processing library 283 11 months ago for ( 🇵🇰)Urdu language

awesome-nlp / NLP in Persian / Libraries
Hazm 1,208 4 months ago Persian NLP Toolkit
Parsivar 230 11 months ago : A Language Processing Toolkit for Persian
Perke 70 about 1 month ago : Perke is a Python keyphrase extraction package for Persian language. It provides an end-to-end keyphrase extraction pipeline in which each component can be easily modified or extended to develop new models
Perstem 18 over 8 years ago : Persian stemmer, morphological analyzer, transliterator, and partial part-of-speech tagger
ParsiAnalyzer 152 7 months ago : Persian Analyzer For Elasticsearch
virastar 82 over 6 years ago : Cleaning up Persian text!

awesome-nlp / NLP in Persian / Datasets
Bijankhan Corpus : Bijankhan corpus is a tagged corpus that is suitable for natural language processing research on the Persian (Farsi) language. This collection is gathered form daily news and common texts. In this collection all documents are categorized into different subjects such as political, cultural and so on. Totally, there are 4300 different subjects. The Bijankhan collection contains about 2.6 millions manually tagged words with a tag set that contains 40 Persian POS tags
Uppsala Persian Corpus (UPC) : Uppsala Persian Corpus (UPC) is a large, freely available Persian corpus. The corpus is a modified version of the Bijankhan corpus with additional sentence segmentation and consistent tokenization containing 2,704,028 tokens and annotated with 31 part-of-speech tags. The part-of-speech tags are listed with explanations in
Large-Scale Colloquial Persian : Large Scale Colloquial Persian Dataset (LSCP) is hierarchically organized in asemantic taxonomy that focuses on multi-task informal Persian language understanding as a comprehensive problem. LSCP includes 120M sentences from 27M casual Persian tweets with its dependency relations in syntactic annotation, Part-of-speech tags, sentiment polarity and automatic translation of original Persian sentences in English (EN), German (DE), Czech (CS), Italian (IT) and Hindi (HI) spoken languages. Learn more about this project at
ArmanPersoNERCorpus 56 over 4 years ago : The dataset includes 250,015 tokens and 7,682 Persian sentences in total. It is available in 3 folds to be used in turn as training and test sets. Each file contains one token, along with its manually annotated named-entity tag, per line. Each sentence is separated with a newline. The NER tags are in IOB format
FarsiYar PersianNER 224 over 3 years ago : The dataset includes about 25,000,000 tokens and about 1,000,000 Persian sentences in total based on . The NER tags are in IOB format. More than 1000 volunteers contributed tag improvements to this dataset via web panel or android app. They release updated tags every two weeks
PERLEX : The first Persian dataset for relation extraction, which is an expert translated version of the “Semeval-2010-Task-8” dataset. Link to the relevant publication
Persian Syntactic Dependency Treebank : This treebank is supplied for free noncommercial use. For commercial uses feel free to contact us. The number of annotated sentences is 29,982 sentences including samples from almost all verbs of the Persian valency lexicon
Uppsala Persian Dependency Treebank (UPDT) : Dependency-based syntactically annotated corpus
Hamshahri : Hamshahri collection is a standard reliable Persian text collection that was used at Cross Language Evaluation Forum (CLEF) during years 2008 and 2009 for evaluation of Persian information retrieval systems

awesome-nlp / NLP in Ukrainian
awesome-ukrainian-nlp 166 about 1 month ago a curated list of Ukrainian NLP datasets, models, etc
UkrainianLT 30 over 2 years ago another curated list with a focus on machine translation and speech processing

awesome-nlp / NLP in Hungarian
awesome-hungarian-nlp 224 about 1 year ago : A curated list of free resources dedicated to Hungarian Natural Language Processing

awesome-nlp / NLP in Portuguese
Portuguese-nlp 240 16 days ago a List of resources and tools developed with focus on Portuguese

awesome-nlp / Other Languages
pymorphy2 1,123 5 months ago Russian: - a good pos-tagger for Russian
ICU Tokenizer Asian Languages: Thai, Lao, Chinese, Japanese, and Korean implementation in ElasticSearch
CLTK 839 3 months ago Ancient Languages: : The Classical Language Toolkit is a Python library and collection of texts for doing NLP in ancient languages
NLPH_Resources 249 about 1 month ago Hebrew: - A collection of papers, corpora and linguistic resources for NLP in Hebrew

Backlinks from these awesome lists:

More related projects: