My current hunch is that they use a lot of non easily accessible online ressources (including a specific archive owned by someone named Anna).

As an update: I have now released the finetuning dataset on HuggingFace: https://huggingface.co/datasets/Pclanglais/MonadGPT

Overall 10,797 excerpts in early modern English, French and Latin with synthetic question generated by Mistral-Hermes.

Well that was actually my original motivation for finetuning. Even GPT-4 is not so good with a proper prompt: the text feels fake and/or struggle to maintain cultural consistency. I think finetuning works better for this task, as there are too many directives to give and it helps to relieve the model from anachronistic RLHF.

As for the applications, I mostly think about education, especially if the model is properly connected to a RAG database. Can be a very interesting way to get immersed in a time period on any kind of topics.

Link to the ongoing demo for MonadGPT, with generous GPU support from HuggingFace : https://huggingface.co/spaces/Pclanglais/MonadGPT

The model has been published as well (and soon the dataset): https://huggingface.co/Pclanglais/MonadGPT?text=Hi.