lorax

Memory optimizer

A JAX transform that simplifies the training of large language models by reducing memory usage through low-rank adaptation.

LoRA for arbitrary JAX models and functions

GitHub

132 stars
4 watching
5 forks
Language: Python
last commit: 9 months ago
Linked from 1 awesome list


Backlinks from these awesome lists:

Related projects:

Repository Description Stars
rentruewang/koila A lightweight wrapper around PyTorch to prevent CUDA out-of-memory errors and optimize model execution 1,821
google/jaxopt An open-source project providing hardware accelerated, batchable and differentiable optimizers in JAX for deep learning. 933
lge-arc-advancedai/auptimizer Automates model building and deployment process by optimizing hyperparameters and compressing models for edge computing. 200
zackzikaixiao/fedgrab A tool for training federated learning models with adaptive gradient balancing to handle class imbalance in multi-client scenarios. 13
jiangoforit/yellowfin_pytorch An optimizer that automatically tunes momentum and learning rate using local quadratic approximation. 287
typemonkey/turtle A programming language designed to optimize memory usage through long-term storage of data 5
dmlc/mxnet-memonger A tool for optimizing deep learning models to reduce memory usage without sacrificing performance 308
mapillary/inplace_abn An optimization technique to reduce memory usage in deep neural networks during training 1,321
jshilong/gpt4roi Training and deploying large language models on computer vision tasks using region-of-interest inputs 506
lyogavin/anima An optimization technique for large language models allowing them to run on limited hardware resources without significant performance loss. 6
alibaba/conv-llava This project presents an optimization technique for large-scale image models to reduce computational requirements while maintaining performance. 104
kvcache-ai/ktransformers A flexible framework for LLM inference optimizations with support for multiple models and architectures 736
google-deepmind/optax A gradient processing and optimization library designed to facilitate research and productivity in machine learning by providing building blocks for custom optimizers and gradient processing components. 1,697
alshedivat/fedpa A modular JAX implementation of federated learning via posterior averaging for decentralized optimization 49
leonardgoeke/anymod.jl A Julia framework for creating large-scale linear optimization models of energy system capacity expansion with multiple periods 70