Hi all, today Nvidia released a new driver version that appears to allow the GPU to use system memory instead of crashing when it runs out, seen here: https://nvidia.custhelp.com/app/answers/detail/a_id/5490/~/system-memory-fallback-for-stable-diffusion
I was wondering if this is compatible with LLM, and how I could enable that (or if it would just work by default).
This works by default and it’s bad. The only way I’d accept it is if the overage is by less than a gig and it attempted to clear off the system ram as fast as possible. Otherwise you may as well not use the GPU at all and take the slow ride.