I’m only getting 0.8 tokens/second with my 3060 12gb using Zephyr 7b beta.
I’ll admit I barely know what I’m doing, but was I wrong to expect a little more? I was hoping for something at least a quarter the speed of gpt-3.5…
You must log in or register to comment.
Sounds like you are executing that with CPU. when you do nvidia-smi, do you see memory and GPU consumption?