There is new rocket 3b that might be worth a try. It’s suspiciously high in benchmarks so i suspect contamination of the dataset, but I saw people have good experience with it.
I was very impressed by rocket 3B in the brief time I tested it. Unfortunately I’ve only ever used mistral based 7B’s so I can’t compare it to older 7B’s but it wouldn’t surprise me if the benchmark results are accurate and it is as good as the old 7B’s.
I’m glad I tried it as now I know to keep an eye on 3B progress. Might not be too long before 3B’s are performing at the level of current mistral 7Bs!
One weird thing though. It crashed for me when I attempted to load in all layers even though I had the VRAM space and when loading 32/35 layers it gave me the same inference speed as when I load 32/35 layers of a 7B.
What inference engine did you use? It’s possibly a bug as these things tend to happen with the new models.
I for one can’t wait for the lookahead decoding in llamacpp and others, combine that with some smaller models and we’ll have blazing fast speeds on pennies worth of hardware from what i recon.
You are probably right about it being a bug as at first I couldn’t get the model to work at all (it crashed koboldccp when loading up) but it was just because I had a week old version of koboldccp. I needed to download the version that came out like 4 days ago (at that time) ha! Then it loaded up fine but with that already mentioned quirk. I guess it will get fixed in short time.
Yeah the future of local LLM’s lies in the smaller models for sure!
There is new rocket 3b that might be worth a try. It’s suspiciously high in benchmarks so i suspect contamination of the dataset, but I saw people have good experience with it.
I was very impressed by rocket 3B in the brief time I tested it. Unfortunately I’ve only ever used mistral based 7B’s so I can’t compare it to older 7B’s but it wouldn’t surprise me if the benchmark results are accurate and it is as good as the old 7B’s.
I’m glad I tried it as now I know to keep an eye on 3B progress. Might not be too long before 3B’s are performing at the level of current mistral 7Bs!
One weird thing though. It crashed for me when I attempted to load in all layers even though I had the VRAM space and when loading 32/35 layers it gave me the same inference speed as when I load 32/35 layers of a 7B.
Thanks for the input.
What inference engine did you use? It’s possibly a bug as these things tend to happen with the new models.
I for one can’t wait for the lookahead decoding in llamacpp and others, combine that with some smaller models and we’ll have blazing fast speeds on pennies worth of hardware from what i recon.
I use koboldccp.
You are probably right about it being a bug as at first I couldn’t get the model to work at all (it crashed koboldccp when loading up) but it was just because I had a week old version of koboldccp. I needed to download the version that came out like 4 days ago (at that time) ha! Then it loaded up fine but with that already mentioned quirk. I guess it will get fixed in short time.
Yeah the future of local LLM’s lies in the smaller models for sure!