I ended up just scrutinizing the server code to understand it better and found that the prompt needs to follow a very specific format or else it won’t work well:
prompt: \A chat between a curious human and an artificial intelligence assistant. The assistant gives helpful, detailed, and polite answers to the human’s questions.\nUSER:[img-12]${message}\nASSISTANT:``
you have all the APIs whats stopping you from putting something like this together? personally for me the only challenge is finding projects compatible with M1 that offer Metal offloading— but for linux it should be relatively straightforward to implement
not prefer it bur recognize its user base— metal + the unified memory have a lot to offer and the compute is there… there just rly no adoption other than a few select projects like llama.cpp and some of the other text-inferencing engines.
I rly wish MPS was more widely adopted by now… hate seeing just CUDA or CPU in all these new libraries
tried coqui and had issues with performance— read online and its doesnt seem to fully support MPS.
for now i’m using upon edge-tts which is doing the trick for now and is pretty decent/free.
is xtts supported on macs?
Update: quick reddit search (which i should’ve done prior to posting tbh) led me to this post: ai_voicechat_script
the licensing on this blows but they have a very unique model IMO: StyleTTS
it picks up the appropriate voice/intonation according to the text which i personally haven’t seen being done yet!
claude is dogshit for code generation from my experience
use LM Studio
damn llama.cpp has a monopoly indirectly 😂
I would still give it a try— it’s misleading to think these coding models are only good at that, being good at coding actually has shown to improve its scores across multiple benchmarks.
Deepseek-Coder has a 1B model I believe that’s outperforming 13B models— I’ll check back once I find a link
Edit: found it https://evalplus.github.io/leaderboard.html
I saw their 7B model closing in on gpt-4 scores in some benchmarks which is absolutely wild but also sus