So Mistral-7b is a pretty impressive 7B param model … but why is it so capable? Do we have any insights into its dataset? Was it trained very far beyond the scaling limit? Any attempts at open reproductions or merges to scale up # of params?
You must log in or register to comment.
Lack of censorship is a key factor as it maximises the predictive abilities of the model.
I second this. Mistral-7B gave me good results. After fine-tuning it’s result is even better.
Mistral-7B gave me good results
Can you expand upon that? Do you mean in terms of its ability to write at a college level without major grammatical errors?
The results are okay, but I’m hard-pressed to call it “very capable”. My perspective on it is that other bigger models are making mistakes they shouldn’t be making because they were “trained wrong”.
I’m guessing GQA helped. Llama2 70b and 34b used Grouped Query Attention but it wasn’t used for Llama2 7/13b.
Knowledge is a strange goal for any model when we have the internet. IMO. Just connect your model to a web search.
My current hunch is that they use a lot of non easily accessible online ressources (including a specific archive owned by someone named Anna).
Why can we get a 20 - 34b version of this very capable Mistral?
Having used it a lot, I can say for sure that without much prompting it readily produces junk web text, urls etc, so it is not a fully filtered or fully synthetic dataset.
My guess would be that it’s just ‘a bit better filtered than llama-2’, and maybe slightly more trained on that set. Slightly better quality set, slightly more trained on that set.
My intuition based on this, is that per parameter size EVERYTHING open source could be optimized considerably more.
It’s simply the time bonus - coming after all the big models.
- better filtering - kill outright junk
- you use already big models (OpenAI and LLama) that you can use for data tuning and filtering
- use available synthetic data
