I’m on my way to deploy a GGUF model on Huggingface space (free hardware CPU and RAM).Currently I’m using a GGUF model because I need to run it using CPU. Later, I have plans to run AWQ models on GPU. I’m currently thinking about ctransformers or llama-cpp-python. Please suggest me which one should I use as a beginner with a plan of integrating llms with websites in future.
Comparison Aspects
Speed
Computational Power Efficiency
Readability and ease of use
Popularity and availability of educational resources
Extra Questions
If I learn ctransformers, is it gonna help me when I will use the huggingface transformers library to load gpu based models in the future? Which one has more resources to solve problems? which one requires less code to run? consider all these aspects and you must choose one between the two
Do I need to learn llama.cpp or C++ to deploy models using llama-cpp-python library?
I used to run AWQ quantized models in my local machine and there is a huge difference in quality. Same model with same bit precision performs much, much worse in GGUF format compared to AWQ. Is there something wrong? Suggest me some fixes
You must log in or register to comment.
“Do I need to learn llama.cpp or C++ to deploy models using llama-cpp-python library?” No its pure python
Also AWQ has entire engines for efficieny, look into aphrodite engine, supposably the fastest for awq
vLLM is way faster, but its pretty barebones and VRAM spikes hard.