Add offloading from VRAM for LLM and limiting GPU usage to long prompts

[alex-ie]

On low VRAM I tend to run with -ngl 1 and still that is often a significant amount of VRAM. Also, I have noted kcpp tends to try to use GPU for generation in such scenarios which does not improve speed much. But -ngl 1 improves long prompts processing greatly vs. CPU only.

How difficult will it be to add option for offloading from VRAM for LLMs similarly how it's done for image generation now? Secondly add another argument to limit using GPU for PP above certain (given in argument or set to reasonable number for efficiency) length (use CPU if shorter) and not use GPU for generation?

Thinking a bit longer, maybe using 2nd argument will activate offloading and that 1st flag is not needed to be added (I see kcpp tends to expose options in both cli and GUI so each adds complexity).

Offloading for LLM will be very beneficial on low VRAM if one wants to keep LLM session and run SD time after time. TIA

[henk717]

The sd style thing is impossible for us when llamacpp doesn't have such a feature. Were still bound to the engines we work with and trying to manually change it that low level would not only be very difficult it would risk us being unable to update.

If you are trying to fit both the LLM and SD maybe look into our autoswap feature that can unload the LLM when you image gen.

[alex-ie]

I understand the point. "autoswap feature" will completely unload all caches of LLM - all closed, only engine is running, correct?

[henk717]

Yeah autoswap starts a second koboldcpp as a proxy with the router mode it needs, and then when you do something that triggers a switch it will kill the one with the engine completely and then create a fresh one in its place. We did it that way since we could never get llamacpp to cleanly unload otherwise.