We’re happy to release GPT-Fast, a fast and hackable implementation of transformer inference in <1000 lines of native PyTorch with support for quantization, speculative decoding, TP, Nvidia/AMD support, and more!

Check out the blog post describing the techniques here: https://pytorch.org/blog/accelerating-generative-ai-2/

And check out the code here: https://github.com/pytorch-labs/gpt-fast

To be clear, this is intended more as a minimal “tutorial” of how you get really good inference performance rather than a library. Hopefully y’all find it useful!

Happy to answer any questions.