BTW, one last thing on my wishlist (in addition to notebook mode) is prompt caching/scrolling.
I realized that the base exllamav2 backend in ooba (and not the HF hack) doesn’t cache prompts, so prompt processing with 50K+ context takes well over a minute on my 3090. I don’t know if that’s also the case in exui, as I did not try a mega context prompt in my quick exui test.
BTW, one last thing on my wishlist (in addition to notebook mode) is prompt caching/scrolling.
I realized that the base exllamav2 backend in ooba (and not the HF hack) doesn’t cache prompts, so prompt processing with 50K+ context takes well over a minute on my 3090. I don’t know if that’s also the case in exui, as I did not try a mega context prompt in my quick exui test.