Parts wise, a threadripper + ASUS Pro WS WRX80E-SAGE SE WiFi II is already a 2k price floor.

each 4090 is 2-2.3k

each 3090 is 1-1.5k

so building a machine from scratch will run you easily 8-10k off 4090’s and 6-8k off 3090’s. If you already have some GPUS or parts you would still problaly need 2 or more extra gpu’s plus the space and power to run them.

to my specific situation i would have to grab the treadripper, mobo, a case, ram, 2 more cards im looking at potentially 5-7k worth of damage. OR… pay 8.6k for a mac pro m2 and get an entire extra machine to play with.

There’s definitely an entire Mac Pro M3 series on the way considering they just released the laptops, it’s only a matter of time for them to shoot out the announcements. So i would definitely feel a bit peeved if i bought the M2 tower only for a month or two later apple to release the m3 versions.

shit, tell me about it. i transitioned here from gaming… i already had a 4090 24gb and was pretty happy with it until decent 70b models came out. then i had to splurge and picked between a 2nd 4090 or a 3090. I went with the 3090 because it’s still just 24gb of vram and the 4090s are a bit fat.

Well turns out i needed to upgrade my as my measly 1k PSU was choking hard. so i upgraded to a 2k psu just to have that extra wiggle room. I quickly ran out of space as well from hoarding data, so i picked up a spare NVME stick. then i learned that my RAM was too low and somehow running large models (70b) requires more ram available, i didn’t even know RAM was necessary for running LLMs if you were using GPUs so i filled up my ram slots.

so all in i spent around 4-5k, the 4090 original build being the bulk of it, but the upgrades with the 3090, ram, psu and nvme wasn’t cheap.

and now…i keep reading about the 120b goliath model, that’s getting rave reviews. and it’s out of reach for me with my 48gb Vram, i9, and 96gb ram. i can’t get it to run on ooga, can’t get it to run on kobold. And im getting real tempted by the new mac products that just came out namely the m3 max versions that have 128gb of unified memory, hell maybe even a mac studio for 5k with 196gb unified memory. would even look at the mac pro tower but thats even more expensive.

OR i could buy 2 more 3090’s a thread ripper, and CPU for it. and squeeze it all into my tower.

either way it’s very expensive to run it locally. I used to think i was at the peak running 70B models, but now the 120b models are starting to show up and i don’t know how to move forward.

So does this fit in 48gb vram or nah?