awesome-bigdata

The world's most popular open source database

The world's most advanced open source database

object-relational database management system

high-performance MPP data warehouse platform

Bistro

general-purpose data processing engine for both batch and stream analytics. It is based on a novel data model, which represents data via and processes data via as opposed to having only set operations in conventional approaches like MapReduce or SQL

platform for distributed processing and real-time analytics. Integrates with many of the popular technologies in the Big Data ecosystem (Kafka, HDFS, Spark, etc.)

framework for distributed processing. Integrates MapReduce (parallel processing), YARN (job scheduling) and HDFS (distributed file system)

Tigon

High Throughput Real-time Stream Processing Framework

Pachyderm is a data storage platform built on Docker and Kubernetes to provide reproducible data processing and analysis

Polyaxon

A platform for reproducible and scalable machine learning and deep learning

Smooks

An extensible Java framework for building XML and non-XML (CSV, EDI, Java, etc...) streaming applications

AddThis Hydra

distributed data processing and storage system originally developed at AddThis

run Spark on Hadoop MapReduce v1

a unified, enterprise platform for big data stream and batch processing

an unified model and set of language-specific SDKs for defining and executing data processing workflows

a simple Java API for tasks like joining and data aggregation that are tedious to implement on plain MapReduce

collection of user-defined functions for Hadoop and Pig developed by LinkedIn

high-performance runtime, and automatic program optimization

real-time big data streaming engine based on Akka

framework for in-memory data model and persistence

BSP (Bulk Synchronous Parallel) computing framework

programming model for processing large data sets with a parallel, distributed algorithm on a cluster

high level language to express data analysis programs for Hadoop

retainable evaluator execution framework to simplify and unify the lower layers of big data systems

framework for stream processing, implementation of S4

framework for in-memory cluster computing

framework for stream processing, part of Spark

framework for stream processing by Twitter also on YARN

stream processing framework, based on Kafka and YARN

application framework for executing a complex DAG (directed acyclic graph) of tasks, built on YARN

abstraction over YARN that reduces the complexity of developing distributed applications

an interface that allows for writing distributed computing programs providing lots of simple, flexible, powerful APIs to easily handle data of any scale

data processing and querying library

High Performance, Custom Data Warehouse on Top of MapReduce

framework for data management/analytics on Hadoop

Damballa Parkour

MapReduce library for Clojure

Datasalt Pangool

alternative MapReduce paradigm

real-time engine is designed to enable distributed, asynchronous, real time in-memory big-data computations in as unblocked a way as possible, with minimal overhead and impact on performance

Hadoop enhancement which removes single point of failure

Map Reduce framework

distributed in-memory datastore

create data pipelines to help themæingest, transform and analyze data

map reduce framework

fault tolerant stream processing framework

platform for distributed processing and real-time analytics. Provides toolkits for advanced analytics like geospatial, time series, etc. out of the box

declarative programming language for working with structured, semi-structured and unstructured data

is a set of libraries, tools, examples, and documentation focused on making it easier to build systems on top of the Hadoop ecosystem

framework for real-time analysis of large datasets

Netflix PigPen

map-reduce for Clojure which compiles to Apache Pig

MapReduce framework developed by Nokia

Distributed computation for the cloud

asynchronous job execution system

Python MapReduce and HDFS API for Hadoop

Ray

A fast and simple framework for building and running distributed applications

multi-tenant distributed metric processing system

Skale

High performance distributed data processing in NodeJS

general purpose cluster computing framework

useful for counting activities of event streams over different time windows and finding the most active one

streamsx.topology

Libraries to enable building IBM Streams application in Java, Python or Scala

Tuktu

Easy-to-use platform for batch and streaming computation, built using Scala, Akka and Play!

Twitter Heron

Heron is a realtime, distributed, fault-tolerant stream processing engine from Twitter replacing Storm

Twitter Scalding

Scala library for Map Reduce jobs, built on Cascading

Twitter Summingbird

Streaming MapReduce with Scalding and Storm, by Twitter

TimeSeries AggregatoR by Twitter

The ultrafast and elastic data processing engine. Big or fast data - no fuss, no Java needed

Ambry

a distributed object store that supports storage of trillion of small immutable objects as well as billions of large objects

a way to store large files across multiple machines

Hadoop's storage layer to enable fast analytics on fast data

formerly FhGFS, parallel distributed file system

software storage platform designed

distributed filesystem

object storage system

distributed filesystem

scalable, highly available storage

GGFS, Hadoop compliant in-memory file system

high-performance distributed filesystem

HDFS-compatible storage in Azure cloud

open-source distributed file system

scale-out network-attached storage file system

Seaweed-FS

simple and highly scalable distributed file system

reliable file sharing at memory speed across cluster frameworks

decentralized cloud storage system

Baidu File System

distributed filesystem

Pilosa

Open source distributed bitmap index that dramatically accelerates queries across multiple, massive data sets

commercial object-oriented database management systems

is an open source massively scalable data store. It requires zero administration

Facebook’s Paxos-like NoSQL database

document oriented datastore over Hadoop

horizontally scalable document-oriented NoSQL data store

Schema-agnostic Enterprise NoSQL database technology

NoSQL cloud database service with protocol support for MongoDB

Document-oriented database system

A transactional, open-source Document Database

document database that supports queries like table joins and group by

distributed key/value store, built on Hadoop

column-oriented distributed datastore, inspired by BigTable

column-oriented distributed datastore, inspired by BigTable

Baidu Tera

an Internet-scale database, inspired by BigTable

evolution of HBase made by Facebook

column-oriented distributed datastore

is a fully managed, schemaless database for storing non-relational data over BigTable

column-oriented distributed datastore, inspired by BigTable

InfiniDB

is accessed through a MySQL interface and use massive parallel processing to parallelize queries

Tephra

Transactions for HBase

real-time, multi-tenant distributed database for Twitter scale

column-oriented distributed datastore written in C++, totally compatible with Apache Cassandra

NoSQL flash-optimized, in-memory. Open source and "Server code in 'C' (not Java or Erlang) precisely tuned to avoid context switching and memory copies."

distributed key/value store, implementation of Dynamo paper

a fast, simple, efficient, and persistent key-value store written natively in Go

Bolt

an embedded key-value database for Go

BTDB

Key Value Database in .Net with Object DB Layer, RPC, dynamic IL and much more

BuntDB

a fast, embeddable, in-memory key/value database for Go with custom indexing and geospatial support

Edis

is a protocol-compatible Server replacement for Redis

ElephantDB

Distributed database specialized in exporting data from Hadoop

distributed time series database

GhostDB

a distributed, in-memory, general purpose key-value data store that delivers microsecond performance at any scale

Graviton

a simple, fast, versioned, authenticated, embeddable key-value store database in pure Go(lang)

GridDB

suitable for sensor data stored in a timeseries

HyperDex

a scalable, next generation key-value and document store with a wide array of features, including consistency, fault tolerance and high performance

is an in-memory key-value data store providing full SQL-compliant data access that can optionally be backed by disk storage

LinkedIn Krati

is a simple persistent data store with very low latency and high throughput

distributed key/value storage system

distributed key-value database by Oracle Corporation

in memory key value datastore

Riak

a decentralized datastore

Storehaus

library to work with asynchronous key value stores, by Twitter

SummitDB

an in-memory, NoSQL key/value database, with disk persistance and using the Raft consensus algorithm

Tarantool

an efficient NoSQL database and a Lua application server

TiKV

a distributed key-value database powered by Rust and inspired by Google Spanner and HBase

Tile38

a geolocation data store, spatial index, and realtime geofence, supporting a variety of object types including latitude/longitude points, bounding boxes, XYZ tiles, Geohashes, and GeoJSON

TreodeDB

key-value store that's replicated and sharded and provides atomic multirow writes

a new generation multi-model graph database for the modern complex data environment

implementation of Pregel, based on Hadoop

implementation of Pregel, part of Spark

multi model distributed database

DGraph

A scalable, distributed, low latency, high throughput graph database aimed at providing Google production level scale and throughput, with low enough latency to be serving real time user queries, over terabytes of structured data

EliasDB

a lightweight graph based database that does not require any third-party libraries

TAO is the distributed data store that is widely used at facebook to store and serve the social graph

GCHQ Gaffer

Gaffer by GCHQ is a framework that makes it easy to store large-scale graphs in which the nodes and edges have statistics

Google Cayley

open-source graph database

graph processing framework

a core C++ GraphLab API and a collection of high-performance machine learning and data mining toolkits built on top of the GraphLab API

resilient Distributed Graph System on Spark

Gremlin

graph traversal Language

Infovore

RDF-centric Map/Reduce framework

tools to construct large-scale graphs on top of Hadoop

open-source, distributed graph database with multiple options for storage backends (Bigtable, HBase, Cassandra, etc.) and indexing backends (Elasticsearch, Solr, Lucene)

Massively Parallel Graph processing on GPUs

Microsoft Graph Engine

a distributed in-memory data processing engine, underpinned by a strongly-typed in-memory key-value store and a general distributed computation engine

graph database written entirely in Java

document and graph database

Phoebus

framework for large scale graph processing

distributed graph database, built over Cassandra

Twitter FlockDB

distributed graph database

A free, open-source template for Microsoft® Excel® 2007, 2010, 2013 and 2016 that makes it easy to explore network graphs

an explanation of what columnar storage is and when you might want it

column-oriented analytic database

an open-source column-oriented database management system that allows generating analytical data reports in real time

a distributed, column-oriented database built for large-scale event collection and analytics

column store database

columnar storage format for Hadoop

purpose-built, dedicated analytic data warehouse that offers a columnar engine as well as a traditional row-based one

is designed to manage large, fast-growing volumes of data and provide very fast query performance when used for data warehouses

A GPU powered big data database, designed for analytics and data warehousing, with ANSI-92 compliant SQL, suitable for data sets from 10TB to 1PB

Google's cloud offering backed by their pioneering work on Dremel

Amazon's cloud offering, also based on a columnar datastore backend

IndexR

an open-source columnar storage format for fast & realtime analytic with big data

LocustDB

an experimental analytics database aiming to set a new standard for query performance on commodity hardware

commercially supported, open-source SQL relational database management system

ActorDB

a distributed SQL database with the scalability of a KV store, while keeping the query capabilities of a relational database

data warehouse service, based on PostgreSQL

BayesDB

statistic oriented SQL database

a simple, modular, networked and distributed transaction layer built atop SQLite

scales out PostgreSQL through sharding and replication

Cockroach

Scalable, Geo-Replicated, Transactional Datastore

Comdb2

a clustered RDBMS built on optimistic concurrency control techniques

distributed database designed to enable scalable, flexible and intelligent applications

distributed database, inspired by F1

distributed SQL database built on Spanner

globally distributed semi-relational database

is an experimental main-memory, parallel database management system that is optimized for on-line transaction processing (OLTP) applications

Haeinsa

linearly scalable multi-row, multi-table transaction library for HBase based on Percolator

NoSQL plugin for MySQL/MariaDB

infinity scalable RDBMS

KarelDB

a relational database backed by Apache Kafka

GPU in-memory database, big data analysis and visualization platform

in memory SQL database witho optimized columnar storage on flash

SQL/ACID compliant distributed database

in-memory, relational database management system with persistence and recoverability

Low-latency, in-memory, distributed SQL data store. Provides SQL interface to in-memory table data, persistable in HDFS

is an in-memory, column-oriented, relational database management system

distributed, realtime, semi-structured database

database used for flexible, high performance analysis of behavioral data

open source software for both file and database synchronization

TiDB

TiDB is a distributed SQL database. Inspired by the design of Google F1

claims to be fastest in-memory database

yugabyteDB

open source, high-performance, distributed SQL database compatible with PostgreSQL

Integrated time series database on top of HBase with built-in visualization, rule-engine and SQL support

a time series storage built to store time series highly compressed and for fast access times

uses MongoDB to store time series data

is a scalable time series database based on Cassandra and Elasticsearch

a time series database with optimised IO and queries, supports pgsql and influx wire protocols

high-performance, open-source SQL database for applications in financial services, IoT, machine learning, DevOps and observability

scalable, general-purpose time series database

Kairosdb

similar to OpenTSDB but allows for Cassandra

a distributed time series database that can be used for storing realtime metrics at long retention

a time series database based on Apache Cassandra

TDengine

a time series database in C utilizing unique features of IoT to improve read/write throughput and reduce space needed to store data

distributed time series database on top of HBase

a time series database and service monitoring system

Beringei

Facebook's in-memory time-series database

an efficient tool for storing and querying series of events

Druid

Column oriented distributed data store ideal for powering interactive applications

Riak TS is the only enterprise-grade NoSQL time series database optimized specifically for IoT and Time Series data

Akumuli

Akumuli is a numeric time-series database. It can be used to capture, store and process time-series data in real-time. The word "akumuli" can be translated from esperanto as "accumulate"

A time-series object store for Cassandra that handles all the complexity of building wide row indexes

Dalmatiner DB

Fast distributed metrics database

Blueflood

A distributed system designed to ingest and process time series data

Timely

Timely is a time series database application that provides secure access to time series data based on Accumulo and Grafana

SiriDB

Highly-scalable, robust and fast, open source time series database with cluster functionality

Thanos

Thanos is a set of components to create a highly available metric system with unlimited storage capacity using multiple (existing) Prometheus deployments

VictoriaMetrics

fast, scalable and resource-effective open-source TSDB compatible with Prometheus. Single-node and cluster versions included

high performance interactive SQL access to all Hadoop data

framework for interactive analysis, inspired by Dremel

table and storage management layer for Hadoop

SQL-like data warehouse system for Hadoop

framework that allows efficient translation of queries involving heterogeneous and federated data

SQL skin over HBase

SQL-like analytic processing for MapReduce

framework for interactive analysis, Inspired by Dremel

SQL-like query language for Cascading

full SQL query engine for big datasets

an open-source, SQL-like Data-as-a-Service Platform based on Apache Arrow

distributed SQL query engine

framework for interactive analysis, implementation of Dremel

Materialize

is a streaming database for real-time applications using SQL for queries and supporting a large fraction of PostgreSQL

SQL engine for online and on-premise use with integrated local data replication and 70+ connectors

an open-source relational database that runs SQL queries continuously on streams, incrementally storing results in tables

SQL-like data warehouse system for Hadoop

database for storing petabyte-scale volumes of structured and semi-structured data

Spark Catalyst

is a Query Optimization Framework for Spark and Shark

Manipulating Structured Data Using Spark

a full-featured SQL-on-Hadoop RDBMS with ACID transactions

interactive query for Hive

distributed data warehouse system on Hadoop

enterprise-class SQL-on-HBase solution targeting big data transactional or operational workloads

A Kafka® replacement for mission critical systems; 10x faster. Written in C++

real-time processing of streaming data at massive scale

serverless fully managed extract, transform, and load (ETL) service

A reverse ETL product that let you sync data from your data warehouse to SaaS Applications. No engineering favors required—just SQL

data collection system

service to manage large amount of log data

distributed publish-subscribe messaging system

Apache NiFi is an integrated data logistics platform for automating the movement of data between disparate systems

Apache Pulsar

a distributed pub-sub messaging platform with a very flexible messaging model and an intuitive client API

tool to transfer data between Hadoop and a structured datastore

open-source bulk data loader that helps data transfer between various databases, storages, file formats, and cloud services

Facebook Scribe

streamed log data aggregator

tool to collect events and logs

Gazette

Distributed streaming infrastructure built on cloud storage which makes it easy to mix and match batch and streaming paradigms

geographically distributed system for joining multiple continuously flowing streams of data in real-time with high scalability and low latency

Heka

open source stream processing software system

HIHO

framework for connecting disparate data sources with Hadoop

distributed message queue system

stream of change capture events for a database

LinkedIn Kamikaze

utility package for compressing sorted integer arrays

LinkedIn White Elephant

log aggregator and dashboard

a tool for managing events and logs

Netflix Suro

log agregattor like Storm and Samza based on Chukwa

Pinterest Secor

is a service implementing Kafka log persistance

Linkedin Gobblin

linkedin's universal data ingestion framework

Skizze

sketch data store to deal with all problems around counting and sketching using probabilistic data-structures

continuous big data ingest infrastructure with a simple to use IDE

data pipeline as a service enabling moving data sources such as MySQL into data warehouses

RudderStack

an open source customer data infrastructure (segment, mParticle alternative) written in go

Zilla

An API gateway built for event-driven architectures and streaming that supports standard protocols such as HTTP, SSE, gRPC, MQTT and the native Kafka protocol

runtime for distributed, and fault tolerant event-driven applications on the JVM

data serialization system

Java libaries for Apache ZooKeeper

OSGi runtime that runs on top of any OSGi framework

framework to build binary protocols

centralized service for process management

a lock service for loosely-coupled distributed systems

Hydrosphere Mist

a service for exposing Apache Spark analytics jobs and machine learning models as realtime, batch or reactive web services

cluster manager

Mara

A lightweight opinionated ETL framework, halfway between plain scripts and Apache Airflow

message passing framework

decentralized solution for service discovery and orchestration

Spotify Luigi

a Python package for building complex pipelines of batch jobs. It handles dependency resolution, workflow management, visualization, handling failures, command line integration, and much more

Spring XD

distributed and extensible system for data ingestion, real time analytics, batch processing, and data export

Twitter Elephant Bird

libraries for working with LZOP-compressed data

asynchronous network stack for the JVM

Apache Airflow

a platform to programmatically author, schedule and monitor workflows

is a service scheduler that runs on top of Apache Mesos

data management framework

workflow job scheduler

cloud-based pipeline orchestration for on-prem, cloud and HDInsight

distributed and fault-tolerant scheduler

Cronicle

Distributed, easy to install, NodeJS based, task scheduler

Dagster

a data orchestrator for machine learning, analytics, and ETL

batch workflow job scheduler

Schedoscope

Scala DSL for agile scheduling of Hadoop jobs

Sparrow

scheduling platform

Cloud-based AzureML, R, Python Machine Learning platform

brain

Neural networks in JavaScript

Oryx

Lambda architecture on Apache Spark, Apache Kafka for real-time large scale machine learning

machine learning library for Cascading

convnetjs

Deep Learning in Javascript. Train Convolutional Neural Networks (or ordinary ones) in your browser

A vectorization and data preprocessing library for deep learning in Java and Scala. Part of the Deeplearning4j ecosystem

Fast, open deep learning for the JVM (Java, Scala, Clojure). A neural network configuration layer powered by a C++ library. Uses Spark and Hadoop to train nets on multiple GPUs and CPUs

Decider

Flexible and Extensible Machine Learning in Ruby

machine learning framework that supports a variety of advanced algorithms, as well as support classes to normalize and process data

text classification with machine learning

Etsy Conjecture

scalable Machine Learning in Scalding

Feast

A feature store for the management, discovery, and access of machine learning features. Feast provides a consistent view of feature data for both model training and model serving

A machine learning platform in Python with a broad collection of ML toolkits, data engineering, and deployment tools

H2O

statistical, machine learning and math runtime with Hadoop. R and Python

Karate Club

An unsupervised machine learning library for graph structured data. Python

Keras

An intuitive neural net API inspired by Torch that runs atop Theano and Tensorflow

Lambdo

Lambdo is a workflow engine which significantly simplifies the analysis process by unifying feature engineering and machine learning operations

Little Ball of Fur

A subsampling library for graph structured data. Python

An Apache-backed machine learning library for Hadoop

distributed machine learning libraries for the BDAS stack

MLPNeuralNet

Fast multilayer perceptron neural network library for iOS and Mac OS X

ML Workspace

All-in-one web-based IDE specialized for machine learning and data science

MOA performs big data stream mining in real time, and large scale machine learning

Text mining made easy. Extract and classify data from text

A matrix library for the JVM. Numpy for Java

nupic

Numenta Platform for Intelligent Computing: a brain-inspired machine intelligence platform, and biologically accurate neural network based on cortical learning algorithms

machine learning server buit on Hadoop, Mahout and Cascading

PyTorch Geometric Temporal

a temporal extension library for PyTorch Geometric

Reinforcement learning for Java and Scala. Includes Deep-Q learning and A3C algorithms, and integrates with Open AI's Gym. Runs in the Deeplearning4j ecosystem

distributed streaming machine learning framework

scikit-learn

scikit-learn: machine learning in Python

Shapley

A data-driven framework to quantify the value of classifiers in a machine learning ensemble

a Spark implementation of some common machine learning (ML) functionality

System for Large Scale Machine Learning at Google

TensorFlow

Library from Google for machine learning using data flow graphs

A Python-focused machine learning library supported by the University of Montreal

A deep learning library with a Lua API, supported by NYU and Facebook

Velox

System for serving machine learning predictions

Vowpal Wabbit

learning system sponsored by Microsoft and Yahoo!

suite of machine learning software

BidMach

CPU and GPU-accelerated Machine Learning Library

micro-benchmarks for testing Hadoop performances

real-world big data workload benchmark

Intel HiBench

a Hadoop benchmark suite

benchmark suite for MapReduce applications

Hadoop cluster benchmarking from Yahoo engineer team

UCSB

extended Yahoo Cloud Serving Benchmark for NoSQL databases

Central security admin & fine-grained authorization for Hadoop

real time monitoring solution

single point of secure access for Hadoop clusters

security module for data stored in Hadoop

BDA

The vulnerability detector for Hadoop and Spark

operational framework for Hadoop mangement

system deployment framework for the Hadoop ecosystem

cluster management framework

cluster manager

Apache Slider

is a YARN application to deploy existing distributed applications on YARN

set of libraries for running cloud services

Cluster manager

library that simplifies application deployment and management

Similar to Apache BigTop based on Groovy language

web application for interacting with Hadoop

multi datacenters replication system

job scheduling and monitoring system

job scheduling and monitoring system

application that can deploy HBase cluster on YARN

a system for automating deployment, scaling, and management of containerized applications

Marathon

Mesos framework for long-running services

Linkis

Linkis helps easily connect to various back-end computation/storage engines

411

an web application for alert management resulting from scheduled searches into Elasticsearch

Adobe spindle

Next-generation web analytics processing with Scala, Spark, and Parquet

a platform that integrates a variety of open source big data technologies in order to offer a centralized tool for security monitoring and analysis

open source web crawler

capturing, processing and sharing of data for NASA's scientific archives

content analysis toolkit

Argus

Time series monitoring and alerting platform

AthenaX

a streaming analytics platform that enables users to run production-quality, large scale streaming analytics using Structured Query Language (SQL)

Atlas

a backend for managing dimensional time series data

open source mobile and web analytics platform, based on Node.js & MongoDB

Run, scale, share, and deploy models — without any infrastructure

Eclipse-based reporting system

ElastAert

ElastAlert is a simple framework for alerting on anomalies, spikes, or other patterns of interest from data in ElasticSearch

Eventhub

open source event analytics platform

open source simulation and visualization platform

Hermes

asynchronous message broker built on top of Kafka

Splunk analytics for Hadoop

Large scale analytics platform by indeed

Web & mobile analytics tool, with data warehouse (AWS, BigQuery) integration

Notebook and project application for interactive data science and scientific computing across all programming languages

data-processing library of an RDBMS to analyze data

Kapacitor

an open source framework for processing, monitoring, and alerting on time series data

open source Distributed Analytics Engine from eBay

PivotalR

R on Pivotal HD / HAWQ and PostgreSQL

Rakam

open-source real-time custom analytics platform powered by Postgresql, Kinesis and PrestoDB

auto-scaling Hadoop cluster, built-in data connectors

SnappyData

a distributed in-memory data store for real-time operational analytics, delivering stream analytics, OLTP (online transaction processing) and OLAP (online analytical processing) built on Spark in a single integrated cluster

Snowplow

enterprise-strength web and event analytics, powered by Hadoop, Kinesis, Redshift and Postgres

R frontend for Spark

analyzer for machine-generated data

cloud based analyzer for machine-generated data

Substation

Substation is a cloud native data pipeline and transformation toolkit written in Go

unified open source environment for YARN, Hadoop, HBASE, Hive, HCatalog & Pig

Search engine library

Search platform for Apache Lucene

Elassandra

is a fork of Elasticsearch modified to run on top of Apache Cassandra in a scalable and resilient peer-to-peer architecture

Search and analytics engine based on Apache Lucene

– Freemium robust web application for exploring, filtering, analyzing, searching and exporting massive datasets scraped from across the Web

continuous indexing system

continuous indexing system

implementation of Percolator, part of HBase

quickly and easily search for any content stored in HBase

is a Faceted Search implementation written purely in Java, an extension to Apache Lucene

LinkedIn Cleo

is a flexible software library for enabling rapid development of partial, out-of-order and real-time typeahead search

search architecture at LinkedIn

LinkedIn Zoie

is a realtime search/indexing system written in Java

MG4J (Managing Gigabytes for Java) is a full-text search engine for large document collections written in Java. It is highly customisable, high-performance and provides state-of-the-art features and new research algorithms

fulltext search engine

is an engine for low-latency computation over large data sets. It stores and indexes your data such that queries, selection and processing over the data can be performed at serving time

Facebook Faiss

is a library for efficient similarity search and clustering of dense vectors. It contains algorithms that search in sets of vectors of any size, up to ones that possibly do not fit in RAM. It also contains supporting code for evaluation and parameter tuning. Faiss is written in C++ with complete wrappers for Python/numpy

Annoy

is a C++ library with Python bindings to search for points in space that are close to a given query point. It also creates large read-only file-based data structures that are mmapped into memory so that many processes may share the same data

Weaviate

Weaviate is a GraphQL-based semantic search engine with build-in (word) embeddings

MySQL databases in Amazon's cloud

evolution of MySQL 6.0

MySQL databases in Google's cloud

enhanced, drop-in replacement for MySQL

MySQL implementation using NDB Cluster storage engine

enhanced, drop-in replacement for MySQL

ProxySQL

High Performance Proxy for MySQL

TokuDB is a storage engine for MySQL and MariaDB

is a collaboration among engineers from several companies that face similar challenges in running MySQL at scale

hybrid of MapReduce and DBMS

high-performance data warehouse appliances

Scalable Open Source PostgreSQL-based Database Cluster

Open Source Recommendation Engine Built Entirely Inside PostgreSQL

open source MPP database system solely targeted at data warehousing and data mart applications

multi-peta-byte database / MPP derived by PostgreSQL

An open-source time-series database optimized for fast ingest and complex queries

The Streaming SQL Database. An open-source relational database that runs SQL queries continuously on streams, incrementally storing results in tables

key/value cache for flash storage

fork of Memcache

Twemproxy

A fast, light-weight proxy for memcached and redis

Twitter Fatcache

key/value cache for flash storage

Twitter Twemcache

fork of Memcache

ACID-compliant DBMS developed by Pervasive Software, optimized for embedding in applications

a software library that provides a high-performance embedded database for key/value data

HanoiDB

Erlang LSM BTree Storage

LevelDB

a fast key-value storage library written at Google that provides an ordered mapping from string keys to string values

ultra-fast, ultra-compact key-value embedded data store developed by Symas

embeddable persistent key-value store for fast storage based on LevelDB

business intelligence platform in the cloud

Blazer

business intelligence made simple

lean business intelligence platform to visualize and explore your data

notebook-based anlytics and visualisation platform using SQL or drag-and-drop

self-service business intelligence tool in the cloud

Large scale geospatial analytics for Google BigQuery based on Kepler.gl

platform for data products and embedded analytics

powerful business intelligence suite

customisable Business Intelligence platform

Interactive Big Data Analytics

Performance Monitoring for Amazon Redshift

Metabase

The simplest, fastest way to get business intelligence and analytics to everyone in your company

business intelligence software and platform

software platforms for business intelligence, mobile intelligence, and network applications

Fast, clean SQL client and business intelligence

business intelligence platform

business intelligence and analytics platform

Open source business intelligence platform, supporting multiple data sources and planned queries

Open source analytics platform

open source business intelligence platform. (former )

modern B.I platform powered by Apache Spark

business intelligence platform

Big Data Analytics

Airpal

Web UI for PrestoDB

fast, simple and flexible JavaScript (HTML5) charting library featuring pure JS API

Arbor

graph visualization library using web workers and jQuery

Banana

visualize logs and time-stamped data stored in Solr. Port of Kibana

Bloomery

Web UI for Impala

A powerful Python interactive visualization library that targets modern web browsers for presentation, with the goal of providing elegant, concise construction of novel graphics in the style of D3.js, but also delivering this capability with high-performance interactivity over very large or streaming datasets

D3-based reusable chart library

CartoDB

open-source or freemium hosting for geospatial databases with powerful front-end editing capabilities and a robust API

responsive, retina-compatible charts with just an img tag

open source HTML5 Charts visualizations

Chartist.js

another open source HTML5 Charts visualization

JavaScript library for exploring large multivariate datasets in the browser. Works well with dc.js and d3.js

Cubism

JavaScript library for time series visualization

JavaScript library for visualizing complex networks

Dimensional charting built to work natively with crossfilter rendered using d3.js. Excellent for connecting charts/additional metadata to hover events in D3

javaScript library for manipulating documents

D3.compose

Compose complex, data-driven visualizations from reusable charts and components

A fairly robust set of reusable charts and styles for d3.js

Dash

Analytical Web Apps for Python, R, Julia, and Jupyter. Built on top of plotly, no JS required

Large scale geospatial analytics for Google BigQuery based on Kepler.gl

High-performance plugin-based React chart for Bootstrap and Material Design

Echarts

Baidus enterprise charts

Envisionjs

dynamic HTML5 visualization

write SQL queries that return SVG charts rather than tables

GitHub-inspired simple and modern SVG charts for the web with zero dependencies

Freeboard

pen source real-time dashboard builder for IOT and other web mashups

Gephi

An award-winning open-source platform for visualizing and manipulating large graphs and network connections. It's like Photoshop, but for graphs. Available for Windows and Mac OS X

simple charting API

graphite dashboard frontend, editor and graph composer

scalable Realtime Graphing

simple and flexible charting API

provides a rich architecture for interactive computing

visualize logs and time-stamped data

open source big data analysis and visualization platform

Matplotlib

plotting with Python

a library built on top of D3 that is optimized for time-series data

chart components for d3.js

Peity

Progressive SVG bar, line and pie charts

Easy-to-use web service that allows for rapid creation of complex charts, from heatmaps to histograms. Upload data to create and style charts with Plotly's online spreadsheet. Fork others' plots

Plotly.js

The open source javascript graphing library that powers plotly

Recline

simple but powerful library for building data applications in pure Javascript and HTML

Redash

open-source platform to query and visualize data

A composable charting library built on React components

a web application framework for R

Sigma.js

JavaScript library dedicated to graph drawing

Superset

a data exploration platform designed to be visual, intuitive and interactive, making it easy to slice, dice and visualize data and perform analytics at the speed of thought

Vega

a visualization grammar

Zeppelin

a notebook-style collaborative data analysis

JavaScript charting library for big data

DataSphere Studio

one-stop data application development management portal

a programming model and micro-kernel style runtime that can be embedded in gateways and small footprint edge devices enabling local, real-time, analytics on the edge devices

Cloud-based bi-directional monitoring and messaging hub

Cloud-based sensor analytics

Platform for Internet of things

Data stream network

Rapid development and connection of intelligent systems

If this then that

Making products smart

NetLytics

Analytics platform to process network data on Spark

Pub/sub messaging platform for IoT

Benchmark of Redshift, Hive, Shark, Impala and Stiger/Tez

Cassandra vs MongoDB vs CouchDB vs Redis vs Riak vs HBase vs Couchbase vs Neo4j vs Hypertable vs ElasticSearch vs Accumulo vs VoltDB vs Scalaris comparison

Guide to monitoring Apache Kafka, including native methods for metrics collection

Guide to monitoring Hadoop, with an overview of Hadoop architecture, and native methods for metrics collection

Guide to monitoring Cassandra, including native methods for metrics collection

- One Trillion Edges: Graph Processing at Facebook-Scale

- Mining of Massive Datasets

- Presto: Distributed Machine Learning and Graph Processing with Sparse Matrices

- MLbase: A Distributed Machine-learning System

- Shark: SQL and Rich Analytics at Scale

- GraphX: A Resilient Distributed Graph System on Spark

- HyperLogLog in Practice: Algorithmic Engineering of a State of The Art Cardinality Estimation Algorithm

- Scalable Progressive Analytics on Big Data in the Cloud

- Druid: A Real-time Analytical Data Store

- Online, Asynchronous Schema Change in F1

- F1: A Distributed SQL Database That Scales

- MillWheel: Fault-Tolerant Stream Processing at Internet Scale

- Scuba: Diving into Data at Facebook

- Unicorn: A System for Searching the Social Graph

- Scaling Memcache at Facebook

- The Unified Logging Infrastructure for Data Analytics at Twitter

- Blink and It’s Done: Interactive Queries on Very Large Data

- Fast and Interactive Analytics over Hadoop Data with Spark

- Shark: Fast Data Analysis Using Coarse-grained Distributed Memory

- Paxos Replicated State Machines as the Basis of a High-Performance Data Store

- Paxos Made Parallel

- BlinkDB: Queries with Bounded Errors and Bounded Response Times on Very Large Data

- Processing a trillion cells per mouse click

- Spanner: Google’s Globally-Distributed Database

- Scarlett: Coping with Skewed Popularity Content in MapReduce Clusters

- Mesos: A Platform for Fine-Grained Resource Sharing in the Data Center

- Megastore: Providing Scalable, Highly Available Storage for Interactive Services

- Finding a needle in Haystack: Facebook’s photo storage

- Spark: Cluster Computing with Working Sets

- Pregel: A System for Large-Scale Graph Processing

- Large-scale Incremental Processing Using Distributed Transactions and Notifications base of Percolator and Caffeine

- Dremel: Interactive Analysis of Web-Scale Datasets

- S4: Distributed Stream Computing Platform

HadoopDB: An Architectural Hybrid of MapReduce and DBMS Technologies for Analytical Workloads

- Chukwa: A large-scale monitoring system

- Dynamo: Amazon’s Highly Available Key-value Store

- The Chubby lock service for loosely-coupled distributed systems

- Bigtable: A Distributed Storage System for Structured Data

- MapReduce: Simplied Data Processing on Large Clusters

- The Google File System

Spark in Motion teaches you how to use Spark for batch and streaming data analytics

LiveVideo tutorial that covers machine learning, Tensorflow, artificial intelligence, and neural networks

Introduction to schema design for data warehouse using the star schema method

LiveVideo tutorial that covers searching, analyzing, and visualizing big data on a cluster with Elasticsearch, Logstash, Beats, Kibana, and more

Data Science at Scale with Python and Dask teaches you how to build distributed data projects that can handle huge amounts of data

Streaming Data introduces the concepts and requirements of streaming and real-time data systems

Storm Applied is a practical guide to using Apache Storm for the real-world tasks associated with processing and analyzing real-time data streams

This comprehensive, hands-on guide combining the fundamental building blocks and emerging research in stream processing is ideal for application designers, system builders, analytic developers, as well as students and researchers in the field

Presents a new paradigm suitable for stream and complex event processing

Unified Log Processing is a practical guide to implementing a unified log of event streams (Kafka or Kinesis) in your business

Kafka Streams in Action teaches you everything you need to know to implement stream processing on data flowing into your Kafka platform, allowing you to focus on getting more from your data without sacrificing time or effort

Big Data teaches you to build big data systems using an architecture that takes advantage of clustered hardware along with new tools designed specifically to capture and analyze web-scale data

& - Spark in Action teaches you the theory and skills you need to effectively handle batch and streaming data using Spark. Fully updated for Spark 2.0

Kafka in Action is a fast-paced introduction to every aspect of working with Kafka you need to really reap its benefits

Fusion in Action teaches you to build a full-featured data analytics pipeline, including document and data search and distributed data clustering

Reactive Data Handling is a collection of five hand-picked chapters, selected by Manuel Bernhardt, that introduce you to building reactive applications capable of handling real-time processing with large data loads--free eBook!

A book about data engineering in general and the Azure platform specifically

Grokking Streaming Systems helps you unravel what streaming systems are, how they work, and whether they’re right for your business. Written to be tool-agnostic, you’ll be able to apply what you learn no matter which framework you choose

– Theory of distributed systems. Include parts about time and ordering, replication and impossibility results

Alessandro Negro. Combine graph theory and models to improve machine learning projects

awesome-awesomeness

Other awesome lists

awesome

Even more lists

list

Another list?

awesome-awesome-awesome

WTF!

awesome-analytics

Analytics

awesome-public-datasets

Public Datasets

awesome-graph-classification

Graph Classification

awesome-network-embedding

Network Embedding

awesome-community-detection

Community Detection

awesome-decision-tree-papers

Decision Tree Papers

awesome-fraud-detection-papers

Fraud Detection Papers

awesome-gradient-boosting-papers

Gradient Boosting Papers

awesome-monte-carlo-tree-search-papers

Monte Carlo Tree Search Papers

awesome-kafka

Kafka