I am talking about this particular model:
https://huggingface.co/TheBloke/goliath-120b-GGUF
I specifically use: goliath-120b.Q4_K_M.gguf
I can run it on runpod.io on this A100 instance with “humane” speed, but it is way too slow for creating long form text.
These are my settings in text-generation-webui:
Any advice? Thanks
That is awesome. What kind of platform do you use for that 3 GPUs setup?
Basically for now an X670E mobo + Ryzen 7 7800X3D.
But if you want the full speed, a server mobo with ton of PCI-E lanes will work better.