I just released the NeuralHermes-2.5-Mistral-7B model, which is a DPO fine-tuned version of OpenHermes-2.5-Mistral-7B. Teknium, the creator of the SFT model, confirmed on Twitter that this version improves benchmark scores in AGIEval, GPT4All, and TruthfulQA.
Take is a simple proof of concept: I used Intel’s orca_dpo_pairs (from neural-chat-7b-v3-1) in a ChatML format, and only trained it for one hour on an A100 (using Goole Colab). But it shows the potential of DPO to boost the performance of SFT models, basically for free. I released all the code so that everyone can easily experiment with it and find better parameters (it shouldn’t be difficult). You can also access the W&B project.
Note that the preference dataset is also entirely synthetic, with preferred answers coming from GPT-4/3.5 and rejected responses coming from Llama 2 13b chat. It’s a very cheap and efficient way to convert an instruction dataset (OpenOrca in this case) into a preference dataset. I wasn’t very successful in my previous experiments with DPO using other datasets, so I think there’s something very interesting with this one. We can easily reproduce this dataset and improve it with other sources.
I just wanted to share these thoughts and experiments with the community. I’m writing an article about DPO and this model is actually a lucky by-product of it. I’ll share it when it’s ready.
If you want to try the model, TheBloke already provided GGUF and AWQ versions of it.
really cool! what do you think about using gpt3.5 as the worst output in the hopes to resurface some extra edge?
Yes, I’d say it’d probably work better than the current approach. If you look at the reward plots on wandb, it feels like the problem is too easy for the model, hence slight improvement.
https://preview.redd.it/xhuyiquojg3c1.png?width=2398&format=png&auto=webp&s=67725747e6cd9254e38728149fb6cea3ba85d71e