On the one hand, I get that this is very much an ‘enthusiast’ sub, and many of you are doing this because you were the type to have a 4090 already.
On the other, as someone interested in LLMs, stable diffusion and AI, I’m not sure if investing in the hardware to run these things locally makes economic sense at all. I spec’d out a damned nice workstation at microcenter the other day and the bill was over $4000. Even the gpu alone was over $1700.
If you take a really sober look at the numbers, how does running your own system make sense over renting hardware at runpod or a similar service? The overall sentiment I get from reading the posts here is that a large majority of users here are using their 3090’s to crank out smut. Hey, no judgement, but do you really think runpod cares what you run as long as it doesn’t put them in legal jeopardy?
A 4090 is $.50/hr on some services. Even if you assumed 10h / wk of usage over like 5 years that’s still probably less than the depreciation and power usage of running it locally.
TLDR: I know some of you are doing this simply ‘because you can’ but the value proposition looks sketchy as an outsider.
shit, tell me about it. i transitioned here from gaming… i already had a 4090 24gb and was pretty happy with it until decent 70b models came out. then i had to splurge and picked between a 2nd 4090 or a 3090. I went with the 3090 because it’s still just 24gb of vram and the 4090s are a bit fat.
Well turns out i needed to upgrade my as my measly 1k PSU was choking hard. so i upgraded to a 2k psu just to have that extra wiggle room. I quickly ran out of space as well from hoarding data, so i picked up a spare NVME stick. then i learned that my RAM was too low and somehow running large models (70b) requires more ram available, i didn’t even know RAM was necessary for running LLMs if you were using GPUs so i filled up my ram slots.
so all in i spent around 4-5k, the 4090 original build being the bulk of it, but the upgrades with the 3090, ram, psu and nvme wasn’t cheap.
and now…i keep reading about the 120b goliath model, that’s getting rave reviews. and it’s out of reach for me with my 48gb Vram, i9, and 96gb ram. i can’t get it to run on ooga, can’t get it to run on kobold. And im getting real tempted by the new mac products that just came out namely the m3 max versions that have 128gb of unified memory, hell maybe even a mac studio for 5k with 196gb unified memory. would even look at the mac pro tower but thats even more expensive.
OR i could buy 2 more 3090’s a thread ripper, and CPU for it. and squeeze it all into my tower.
either way it’s very expensive to run it locally. I used to think i was at the peak running 70B models, but now the 120b models are starting to show up and i don’t know how to move forward.