Dear friends,
I decided to write because many are active on HuggieFace with their AI models.
I have been continuously testing AI Models 8/10 hours a day for a year now. And when I say that I test the models, I don’t mean like many do on YouTube to get likes, with type tests. Tell me what the capital of Australia is or tell me who the tenth president of the United States is. Because these tests depress me as well as making me smile. Already 40 years ago my Commodore Vic 20 answered these questions in BASIC language!
I test models very seriously. Being a history buff, my questions are very oriented towards history, culture, geography, literature. So my tests are to try in every way to extrapolate answers and summaries to the AI models.
Now I note with great sadness that models are trained with a lot of data, but there is not enough focus on ensuring that the algorithm is able to extrapolate the data and return it to the user in a faithful and coherent manner.
Now if we want to use templates just to play with creative invented stories like poetry everything can be fine, but when we get serious Open Source templates to be installed locally seem very insufficient to me.
Furthermore, I note that the models are never accompanied in an excellent manner by configuration or preset data which the user often has to try to understand by making various calibrations.
Another issue, the models are always generic, there is no table of models with their actual capabilities.
More guidance would be needed. example This is a model that is good for Medicine, this has been trained with History data etc.
While we find ourselves researching Huggingface in an almost haphazard manner, not to mention total disarray
In Pavero words I want to tell you, since you work hard, you too should ensure that the models, in addition to being filled with data, can then be able to use them and give them to the user.
So let’s take a step forward and improve the progress.
Claudio from Italy
Maybe you overbought (like most of us), the “AI” idea. The models have in some random ways compressed the internet, more or less, and then try to decompress it. As their own warnings say, you’re most likely to get out what you most often put in, so you’re only guaranteed to get out the basics that are repeated a million times, everything else is a game of chances. Now, the reason they have their various benchmarks is, they cannot really evaluate the way you’re trying to evaluate, with your brain. Nor can they predict how to make their models better, not even for their own benchmarks. I’d say it is common knowledge that the kind of “thinking” you’re looking for is something that has just started to happen with tools on top of LLMs.
And one last thing the average consumer has not understood about the benchmarks: when their own tests move from, let’s say, 74% to 75%, and there’s no real pattern to how they do it, maybe they tried 10 different times and 9 times it went to 73%, but they only show us the one attempt that was lucky. So basically, when they move higher and higher % in their tests, they’re also committing the ancient sin of “overfitting”, this process of training and finetuning, rinse and repeat, ends up answering questions “for the wrong reasons”, but they don’t care as long as they can show their boss, or the press, some better %. So the models might move from 75% to 85% in their benchmarks and you might get even less of what you’re looking for. Implied in what I wrote is that we need better tools to look into explainable models, and try to weed out the bad explanations with our brains!
All models are designed on a very similar way (fine tuned ones are even on a top level in this regard) if you are expecting Jarvis like model than I am sorry because I think Tony Stark didn’t born yet 😅 Try having another perspective about the models and its limitations and you could actually start being comfortable with what open source give to our community.