Hey All,
what does making a model prediction look like for your current projects? Are you building a model for a web-app and you’re running on-demand inference? Are you working on a research project or doing some analysis that requires making hundreds of thousands to millions of predictions all at once?
I’m currently at a crossroads with a developer tool I’m building and trying to figure out which types of inference workflows I should be focused on. A few weeks back I posted a tutorial on running Mistral-7B on hundreds of GPUs in the cloud in parallel. I got a decent amount of people saying that batch inference is relevant to them but over the last couple of days I’ve been running into more and more developers that are building web-apps that don’t need to make many predictions all at once. If you were me where would you direct your focus?
Anyways, I’m kinda rambling but I would love to know what you guys are working on and get some advice on the direction I should pursue.
This is really useful feedback, I’d definitely be able to produce a revenue generating product faster if I focus on chatbots… so in terms of trying to get funding for this idea that seems to be the better avenue. In the future I could definitely address both use cases but trying not to spread myself too thin at the moment.