I am running a LLaMa 13B instance (via GPT4all) and am finding inference times to be quite slow, especially for summarization. Does anyone have recommendations for models that can do summarization of 4k+ tokens extremely quickly?
You must log in or register to comment.
orca-mini-3b is good at fast summarizations, but it lies a lot, so ymmv.
you can try to go down to 7b, it will be slightly faster.