• 1 Post
  • 18 Comments
Joined 1 year ago
cake
Cake day: October 30th, 2023

help-circle





  • heheh i can’t read that any more… i really have become very prejudiced when comes to that… to be honest, when it comes to any comparison with GPT-4.

    People have really to understand that even GPT-4 has been aligned, lobotomized and it has been massively downgraded in terms of its perfomance – due to security reasons (what is understandable for me), but anyway this thing still is an absolute beast. if we consider all the restrictions GPT-4 has to undergo, all the smartness at openAI, all the ressources at microsoft and so on, we have to realize that currently nothing is really comparable to GPT-4. Especially not 7B models.



  • That’s a pretty good idea! thanks for your input. I will definitely make a note of it as an issue in my repo and see what I can do.

    Thank you for saying that. It makes me feel valued for my work. I’ve already made a pull request and Gerganov seems to like the work in general, so he would accept a merge. I still need to fix a few things here and there though - the requirements at the llama.cpp dudes are very high : D (but i don’t expect anything else there heheh)





  • Ah one sidenote: selecting a model via dialog is absolutely not intuitive. If you want to navigate into a folder, you have to press space two times. Do not press enter until you decide to choose a specific folder. It doesnt matter that much if you are in parent folders, since the script will search recursively - but of course if you have many files it could take a long time.



  • Hmm, would it really be more expensive? vast.ai can be extremely cheap.

    But I would also be interested in the topic as a whole. I think you would first have to calculate this very precisely. e.g. what is the scope of an average user request? Not many users will fill the entire context window with every request. If we had or could estimate an average value, we would derive how many tokens/second an economically efficient GPU works with and extrapolate that to the price of 1 million tokens.

    But there are other important factors as well:

    • Which country is the hardware located in? Electricity prices can also be extremely different from country to country.

    • How much did the operator have to pay for all his hardware? As a bulk buyer, you almost always get better prices, regardless of the sector.

    • Does he perhaps also operate his own photovoltaic systems and, if so, to what extent?

    • It is also important to remember that not every product leads directly to financial profits. If you have enough capital and can afford it over a certain period of time, you may deliberately consider doing a loss-making business in order to eliminate the competition. This way one could hope to gain reach and popularity with customers, who would then buy other products in the future. (See OpenAI and ChatGPT).


  • I use various things, regularly testing if one of them has become better etc.

    • Mainly llama.cpp backend and server as UI - it has everything what I need, it’s lightweight, it’s hackable

    • Ollama - Simplifies many steps, has very convenient functions and an overall coherent and powerful ecosystem. Mostly in terminal, but sometimes in a modified Ollama Webui

    • Sometimes Agnai and/or RisuAI - nice and powerful UIs with satisfying UXs, however not as powerful as sillytavern. But sillytavern is too much if you are not a RP power-user.

    • My own custom Obsidian ChatGPT-MD + Canvas Chat Addon Addons with local endpoints.

    In general I try to avoid everything that comes with python code and I prefer solutions with as minimal dependencies as possible, so it’s easier to hack and customize to my needs.