I don’t think so, this is something you do when you’re GPU poor, closedai would just not undertrain their models in the first place.

The results are okay, but I’m hard-pressed to call it “very capable”. My perspective on it is that other bigger models are making mistakes they shouldn’t be making because they were “trained wrong”.

I’ve not heard of text-generator.io, is it as performant as vllm on multibatch or is it a wrapper around it?