I recently got a 32GB M1 Mac Studio. I was excited to see how big of a model it could run. It turns out that’s 70B. It is a Q3_K_S model so the 2nd smallest for 70B in GGUF format, but still it’s a 70B model.
As many people know, the Mac shouldn’t be able to dedicate that much RAM to the GPU. Apple limits it to 67%, which is about 21GB. This model is 28GB. So it shouldn’t fit. But there’s a solution to that thanks to these smart people here.
https://github.com/ggerganov/llama.cpp/discussions/2182
They wrote a program to patch that limit in the kernel. You can set it to anything you want. So I cranked mine up to 92%. I also do these couple of things to save RAM.
-
I don’t use the GUI. Just simply logging in and doing nothing uses a fair amount of RAM. I run my Mac headless. I ssh in.
-
I stopped the mds_stores process from running. I saw that it was using up between 500MB and 1GB of RAM. Its the processes that indexes the drives for faster search. Considering my drive is 97% empty, I don’t know what it was doing to use up 1GB of RAM. I normally turn off indexing on all my machines always.
With all that set, the highest I’ve seen in use memory is 31.02GB while running a 70B Q3_K_S model. So there’s headroom. There maybe a lot more. Since my goal is to not swap. I noticed that when I log into the GUI while it’s running a model, the compressed RAM goes up to around 750MB but it still doesn’t swap. So I wonder how far memory compression would let me stretch it. I do notice that it’s not as snappy. With no GUI login, the model runs right away after the model is cached after the first run. With a GUI login, it pauses for a few seconds.
As for performance, it’s 14 t/s prompt and 4 t/s generation using the GPU. It’s 2 and 2 using the CPU. Power consumption is remarkably low. Using the GPU, powermetrics reports 39 watts for the entire machine but my wall monitor says it’s taking 79 watts from the wall. Using the CPU powermetrics reports 36 watts and the wall monitor says 63 watts. I don’t know why it’s so much more efficient at the wall between GPU and CPU. It’s only a 3 watt difference in the machine but 16 watts at the wall.
All in all, I’m super impressed. The M1 32GB Studio may be the runt of the Mac Studio lineup but considering that I paid about what a used 3090 costs on ebay for a new one, I think it’s the best value for performance I have to run LLMs. Since I plan on running this all out 24/7/365, the power savings alone compared to anything else with a GPU will be several hundreds of dollars a year.
That’s where context shifting comes into play. You don’t re-evaluate the entire context. You just process the additions.
“Previously, we had to re-evaluate the context when it becomes full and this could take a lot of time, especially on the CPU. Now, this is avoided by correctly updating the KV cache on-the-fly:”
https://github.com/ggerganov/llama.cpp/pull/3228