Reuters is reporting that OpenAI achieved an advance with a technique called Q* (pronounced Q-Star).

So what is Q*?

I asked around the AI researcher campfire and…

It’s probably Q Learning MCTS, a Monte Carlo tree search reinforcement learning algorithm.

Which is right in line with the strategy DeepMind (vaguely) said they’re taking with Gemini.

Another corroborating data-point: an early GPT-4 tester mentioned on a podcast that they are working on ways to trade inference compute for smarter output. MCTS is probably the most promising method in the literature for doing that.

So how do we do it? Well, the closest thing I know of presently available is Weave, within a concise / readable Apache licensed MCTS lRL fine-tuning package called minihf.

https://github.com/JD-P/minihf/blob/main/weave.py

I’ll update the post with more info when I have it about q-learning in particular, and what the deltas are from Weave.

  • georgejrjrjrOPBEnglish
    1·
    1 year ago

    Edits aren’t working for me somehow, here’s my update:

    First, as I mentioned on twitter but failed to address here, this is at least excellent PR. So that may be all it is, basically a more sophisticated “AGI achieved internally” troll. I would suggest taking Q* discourse with all due salt.

    From context and the description, it looks like OpenAI published about the technique in question here: https://openai.com/research/improving-mathematical-reasoning-with-process-supervision

    The result is pretty unsurprising: given process supervision (i.e., help from a suitably accurate model of a particular process), models perform better.

    Well…yeah. It’s probably an impactful direction for AI as people find ways to build good process models, but it isn’t an especially novel finding, nor is it a reason to blow up a company. This updates me further in the direction of, “Q* discourse was a brilliant PR move to capitalize off of the controversy and direct attention away from the board power struggle.”

    Which doesn’t mean it can’t also be a good intuition pump for the open source world. Every big lab seems to be thinking about model-based supervision, it would be a little bit silly if we weren’t. So coming back to the original question:

    How might we use this?

    I think the question reduces to, “What means of supervision are available?”

    Once you have a supervisor to play “warmer / colder” with the model, the rest is trivial.

    I’m curious what models you all expect to come online to supervise llms. Arithmetic has already been reported. Code, too.

    What else?

  • honestduaneBEnglish
    1·
    1 year ago

    Q* was completely explained, and openAI explained what it was. I was even able to make a YouTube video about it because they’re explanation was so clear, so I was able to explain it as if you were five years old.

    I don’t understand how people believe this is a secretive thing and I don’t understand why people aren’t talking about how simple it is.

    Everybody is talking about this like it’s some grand secret, why?

    I mean, the algorithm is expensive to run, but it’s not that hard to understand.

    Can somebody please explain why everybody’s acting like this is such a big secret thing?

  • sprectzaBEnglish
    1·
    1 year ago

    Yeah I think its MCTS reinforcement learning algorithm. I think DeepMind is the best lab when it comes to depeloping strategy and planning capable agents, given how good AlphaZero and AlphaGo is, and if they integrate it with the “Gemini” project, they really might just “ecliplse” GPT-4. I don’t know how scalable it would be in terms of inference given the amount of compute required.

  • olddoglearnsnewtrickBEnglish
    1·
    1 year ago

    It’s a silicon based version of Qanon. I will be terminated by telling you but wait 'till they launch MAGA (Machine Augmented General AI) !!!

  • wind_dudeBEnglish
    1·
    1 year ago

    is there something other than the letter Q making you think it’s Q-learning?