I’m not really talking about a cryptocurrency, more of an anonymous collaborative system where your local copy of the blockchain syncs up to the most recent stage of the model’s training and dataset, and the process of mining is a pooled effort to find the next best loss reduction.
Training data would obviously be a problem, but I’m sure there are ways to append data to the dataset in the mining process.
Something decentralized, massive, and unstoppable.
Man I’m high…Time to go talk to Mistral.
This isn’t an original concept but the data transfer rates make distributed training like this pretty impractical. Unless you’re doing some galaxy brained techniques you will need to distribute like 120gb every step (for a 70B model) and training on an old step is a waste of time. So parallelizing it on internet connected machines is a horrible option.
You don’t really need this setup anyways as you can train a 4bit 13B Lora on a gaming pc. This would only be useful for big foundation models, maybe, in which case you’re going to get way faster results by just renting some cloud gpus