I’m still new to this and I thought that 128gb CPU ram would be enough to run a 70b model? I also have an RTX 4090. However, everytime I try to run lzlv_Q4_K_M.gguf in Text Generation UI, I get “connection errored out”. Could there be a setting that I should tinker with?
You must log in or register to comment.
It should work with those specs. Not sure what “connection” it means. Perhaps post a screenshot of the console?