Text Gen Web UI. Works great on Mac. I use ggufs, since Llamacpp supports metal.
damn llama.cpp has a monopoly indirectly 😂
It’s just easier to run (and deploy!) cross platform compiled code than to setup 10 different python envs and cross your fingers that it might work this time.
Text Generation UI as the backend and sillytavern as the front end.
KoboldCPP where proper transformers/cuda isn’t supported.
Yep pretty good combo! I also use ooba+Silly and for internet query and pdf ingestion I use LolLLMs Great stuff!
Don’t forget exui: https://github.com/turboderp/exui
Once it implements notebook mode, I am probably going to switch to that, as all my reasons for staying on text gen ui (the better samplers, notebook mode) will be pretty much gone, and (as said below) text gen ui has some performance overhead.
I just switched to KoboldCpp from Text Geb UI 2 days ago.
The OpenAI extension wouldn’t install for me and it was causing issues with SillyTavern which I use as a frontend.
I’m actually really happy now that I’ve switched.
KoboldCpp is so simple is great. I’ve written a simple batch file to launch both KoboldCpp and SillyTavern. All I have to do if I want to try a new model is edit the part of the batch pointing to the name of the model and it just works.
On top of that I can load more layers onto my GPU with KoboldCpp than Text Gen UI so I’m getting faster speeds.
I use sillytavern along with text-generation-webui in api mode. Best setup for roleplay imo.
ST. By far the most customizability.
I used to use Text Generation Web UI, but I changed to KoboldCpp because it’s more lightweight. Besides, I realized I didn’t use all the features of the textgen UI. KoboldCpp as the backend and SillyTavern as the frontend when I want to chat. KoboldCpp alone when I want to play with models by creating stories or something.
Koboldcpp because that only one that work for me right now.
I use various things, regularly testing if one of them has become better etc.
-
Mainly llama.cpp backend and server as UI - it has everything what I need, it’s lightweight, it’s hackable
-
Ollama - Simplifies many steps, has very convenient functions and an overall coherent and powerful ecosystem. Mostly in terminal, but sometimes in a modified Ollama Webui
-
Sometimes Agnai and/or RisuAI - nice and powerful UIs with satisfying UXs, however not as powerful as sillytavern. But sillytavern is too much if you are not a RP power-user.
-
My own custom Obsidian ChatGPT-MD + Canvas Chat Addon Addons with local endpoints.
In general I try to avoid everything that comes with python code and I prefer solutions with as minimal dependencies as possible, so it’s easier to hack and customize to my needs.
-
A bit related. I think all the tools mentioned here are for using an existing UI.
But what if you wanted to easily roll your own, preferably in Python. I know of some options:
Gradio https://www.gradio.app/guides/creating-a-custom-chatbot-with-blocks
Panel https://www.anaconda.com/blog/how-to-build-your-own-panel-ai-chatbots
Reflex (formerly Pynecone) https://github.com/reflex-dev/reflex-chat https://news.ycombinator.com/item?id=35136827
Solara https://news.ycombinator.com/item?id=38196008 https://github.com/widgetti/wanderlust
I like streamlit (simple but not very versatile) And reflex seems to have a richer set of features.
My questions - Which of these do people like to use the most? Or are the tools mentioned by OP also good for rolling your own UI on top of your own software ?
Is anyone working on local UI interested in forming a community of builders? I think it would be great to share knowledge, learn from each other, and ultimately raise the bar for a better UI for everyone.
Anyone who wants to take the lead on this is more than welcome to. I’m just putting the idea out there.