sparse_attention
Attention library
Provides primitives for sparse attention mechanisms used in transformer models to improve computational efficiency and scalability
Examples of using sparse attention, as in "Generating Long Sequences with Sparse Transformers"
2k stars
43 watching
192 forks
Language: Python
last commit: about 6 years agoRelated projects:
| Repository | Description | Stars |
|---|---|---|
| openai/finetune-transformer-lm | This project provides code and model for improving language understanding through generative pre-training using a transformer-based architecture. | 2,167 |
 hiroyuki-kasai/sparsegdlibrary | A collection of unconstrained optimization algorithms for sparse modeling in MATLAB | 53 |
 microsoft/focal-transformer | A vision transformer architecture that uses a novel attention mechanism to capture local-global interactions in images | 547 |
| openai/lm-human-preferences | Training methods and tools for fine-tuning language models using human preferences | 1,240 |
 neuralmagic/sparseml | Enables the creation of smaller neural network models through efficient pruning and quantization techniques | 2,083 |
 huyz1117/bam | An implementation of the Bottleneck Attention Module in TensorFlow using attention mechanism | 12 |
 koichiro11/residual-attention-network | An image classification neural network implementation using attention mechanisms and residual learning | 94 |
| openai/generating-reviews-discovering-sentiment | Generates reviews and discovers sentiment using a language model | 1,512 |
 tusimple/sparse-structure-selection | Re-implements sparse structure selection algorithm for deep neural networks in a modified MXNet framework. | 87 |
 pistony/residualattentionnetwork | A Gluon implementation of Residual Attention Network for image classification tasks | 108 |
| openai/simple-evals | Evaluates language models using standardized benchmarks and prompting techniques. | 2,059 |
 64bit/async-openai | A Rust library providing an asynchronous interface to the OpenAI API | 1,202 |
 macpaw/openai | An OpenAI API wrapper for Swift applications | 2,149 |
 yaroslavyaroslav/openai-sublime-text | An AI-powered Sublime Text plugin for generating text and answering questions using OpenAI models | 208 |
| openai/openai-openapi | A specification for an API providing access to AI capabilities | 1,332 |