LocalLLaMA@poweruser.forumEnglish · 11 months ago

QuIP#: SOTA 2-bit quantization method, now implemented in text-generation-webui (experimental)

6

1

QuIP#: SOTA 2-bit quantization method, now implemented in text-generation-webui (experimental)

LocalLLaMA@poweruser.forumEnglish · 11 months ago

6

Add QuIP# support by oobabooga · Pull Request #4803 · oobabooga/text-generation-webui

QuIP# is a novel quantization method. Its 2-bit performance is better than anything previously available. Repository: https://github.com/Cornell-RelaxML/quip-sharp Blog post: https://cornell-relaxm...

Chat

a_beautiful_rhindB
link
fedilink
English
arrow-up
1·
11 months ago
From the issue about this in the exllamav2 repo, quip was using more memory and slower than exl. How much context can you fit?