DeepSpeed

DeepSpeed
Original author	Microsoft Research
Developer	Microsoft
Initial release	May 18, 2020 (2020-05-18)
Stable release	v0.16.5 / March 27, 2025 (2025-03-27)
Repository	github.com/microsoft/DeepSpeed
Written in	Python, CUDA, C++
Engine
Type	Software library
License	Apache License 2.0
Website	deepspeed.ai

DeepSpeed is an open source deep learning optimization library for PyTorch.

Library

The library is designed to reduce computing power and memory use and to train large distributed models with better parallelism on existing computer hardware. DeepSpeed is optimized for low latency, high throughput training. It includes the Zero Redundancy Optimizer (ZeRO) for training models with 1 trillion or more parameters. Features include mixed precision training, single-GPU, multi-GPU, and multi-node training as well as custom model parallelism. The DeepSpeed source code is licensed under MIT License and available on GitHub.

The team claimed to achieve up to a 6.2x throughput improvement, 2.8x faster convergence, and 4.6x less communication.

See also

• Comparison of deep learning software
• Deep learning
• Machine learning
• TensorFlow

External links

• AI at Scale - Microsoft Research
• GitHub - microsoft/DeepSpeed
• ZeRO & DeepSpeed: New system optimizations enable training models with over 100 billion parameters - Microsoft Research