Local LLMs are wonderful, and we all know that, but something that’s always bothered me is that nobody in the scene seems to want to standardize or even investigate the flaws of the current sampling methods. I’ve found that a bad preset can make a model significantly worse or golden depending on the settings.
It might not seem obvious, or it might seem like the default for whatever backend is already the ‘best you can get’, but let’s fix this assumption. There are more to language model settings than just ‘prompt engineering’, and depending on your sampler settings, it can have a dramatic impact.
For starters, there are no ‘universally accepted’ default settings; the defaults that exist will depend on the model backend you are using. There is also no standard for presets in general, so I’ll be defining the sampler settings that are most relevant:
- Temperature
A common factoid about Temperature that you’ll often hear is that it is making the model ‘more random’; it may appear that way, but it is actually doing something a little more nuanced.
A graph I made to demonstrate how temperature operates
What Temperature actually controls is the scaling of the scores. So 0.5 temperature is not ‘twice as confident’. As you can see, 0.75 temp is actually much closer to that interpretation in this context.
Every time a token generates, it must assign thousands of scores to all tokens that exist in the vocabulary, and the temperature simply helps to either reduce (lowered temp) or increase (higher temp) the scoring of the extremely low probability tokens.
In addition to this, when Temperature is applied matters. I’ll get into that later.
- Top P
This is the most popular sampling method, which OpenAI uses for their API. However, I personally believe that it is flawed in some aspects.
Unsure of where this graph came from, but it’s accurate.
With Top P, you are keeping as many tokens as is necessary to reach a cumulative sum.
But sometimes, when the model’s confidence is high for only a few options (but is divided amongst those choices), this leads to a bunch of low probability options being considered. I hypothesize this is a smaller part of why models like GPT4, as intelligent as they are, are still prone to hallucination; they are considering choices to meet an arbitrary sum.
Top K is doing something even more linear, by only considering as many tokens are in the top specified value, so Top K 5 = only the top 5 tokens are considered always. I’d suggest just leaving it off entirely if you’re not doing debugging.
So, I created my own sampler which fixes both design problems you see with these popular, widely standardized sampling methods: Min P.
What Min P is doing is simple: we are setting a minimum value that a token must reach to be considered at all. The value changes depending on how confident the highest probability token is.
So if your Min P is set to 0.1, that means it will only allow for tokens that are at least 1/10th as probable as the best possible option. If it’s set to 0.05, then it will allow tokens at least 1/20th as probable as the top token, and so on…
“Does it actually improve the model when compared to Top P?” Yes. And especially at higher temperatures.
No other samplers were used. I ensured that Temperature came last in the sampler order as well (so that the measurements were consistent for both).
You might think, "but doesn’t this limit the creativity then, since we are setting a minimum that blocks out more uncertain choices?"Nope. In fact, it helps allow for more diverse choices in a way that Top P typically won’t allow for.
Let’s say you have a Top P of 0.80, and your top two tokens are:
- 81%
- 19%
Top P would completely ignore the 2nd token, despite it being pretty reasonable. This leads to higher determinism in responses unnecessarily.
This means it’s possible for Top P to either consider too many tokens or too little tokens depending on the context; Min P emphasizes a balance, by setting a minimum based on how confident the top choice is.
So, in contexts where the top token is 6%, a Min P of 0.1 will only consider tokens that are at least 0.6% probable. But if the top token is 95%, it will only consider tokens at least 9.5% probable.
0.05 - 0.1 seems to be a reasonable range to tinker with, but you can go higher without it being too deterministic, too, with the plus of not including tail end ‘nonsense’ probabilities.
- Repetition Penalty
This penalty is more of a bandaid fix than a good solution to preventing repetition; However, Mistral 7b models especially struggle without it. I call it a bandaid fix because it will penalize repeated tokens even if they make sense (things like formatting asterisks and numbers are hit hard by this), and it introduces subtle biases into how tokens are chosen as a result.
I recommend that if you use this, you do not set it higher than 1.20 and treat that as the effective ‘maximum’.
Here is a preset that I made for general purpose tasks.
I hope this post helps you figure out things like, “why is it constantly repeating”, or “why is it going on unhinged rants unrelated to my prompt”, and so on.
There’s a lot more I could write about, and I’m also going to write a proper research paper on this. I mainly wanted to share this because I thought it was severely underlooked.
Anyways, I hope this post helps people figure out questions like, “why does this preset work better for me?” or “what do these settings even do?”. I’ve been talking to someone who does model finetuning who asked about potentially standardizing settings + model prompt formats in the future and getting in talks with other devs to make that happen.
The more ‘experimental’ samplers I have excluded from this writeup, as I personally see no benefits when using them. These include Tail Free Sampling, Typical P / Locally Typical Sampling, and Top A (which is a non-linear version of Min P, but seems to perform worse in my subjective opinion). Mirostat is interesting but seems to be less predictable and can perform worse in certain contexts (as it is not a ‘context-free’ sampling method).
Large language models learn deep patterns. Most notably, they target patterns that are not immediately obvious to humans reading the text they create. If the pattern of the long term context implies that the text tends to be repetitive or high-confidence in the abstract, because of deterministic / greedy sampling being used, it will slowly drift towards that repetition over time. And eventually, it will become so focused on this deeper pattern that it’ll be unable to find a way out.
So the main goal of sampling optimization is, we offset that drifting behavior (present in all llm models?), breaking down repetition loops normally formed in the OG sampling. (greedy decoding)
If we assumed the reasoning abilities of a model depend on it not going into repetition loops, maybe this is why larger parameter models are better, Each sampling step has a larger, diverse pool of tokens to choose from.