I’m currently exploring the deployment of Llama models in a production environment and I’m keen to hear from anyone who has ventured into this territory. My primary concern is managing multiple concurrent users while optimizing resources effectively.

While there are numerous methods to tweak Llama for testing with a single user, scaling up poses its own set of challenges. I’m particularly interested in learning how others have approached this problem.I’m curious about projects like vLLM and Huggingface TGI for faster inference. Has anyone had experience with these, and how have they contributed to your scaling efforts?

My goal is to implement an API utilizing Llama models for a small organization’s private use. I’m eager to learn from your experiences and any advice or insights you can share on this topic.