Hi all, today Nvidia released a new driver version that appears to allow the GPU to use system memory instead of crashing when it runs out, seen here: https://nvidia.custhelp.com/app/answers/detail/a_id/5490/~/system-memory-fallback-for-stable-diffusion
I was wondering if this is compatible with LLM, and how I could enable that (or if it would just work by default).
Yes, it’s system wide. You can set your prefered way in Nvidia control panel->global settings-> cuda systemem fallback policity.
Driver default is prefer systemmem fallback, which means it’s going to offload to RAM instead of crashing when VRAM is full.
No System Mem fallback is basically the old memory management, it crashes once your VRAM is full.
This works by default and it’s bad. The only way I’d accept it is if the overage is by less than a gig and it attempted to clear off the system ram as fast as possible. Otherwise you may as well not use the GPU at all and take the slow ride.