Hello!

By popular demand I am planning a fine-tune of https://huggingface.co/dreamgen/opus-v0-7b on top of Yi-34B and wonder whether to use the 200K as the base.

The regular Yi-34B seems slightly better than Yi-34B-200K on standard benchmarks, but I wonder how it “feels” and whether the loss of performance on short context is worth it, given that the regular version can be used up to 32K tokens.

(Yi-34B vs Yi-34B-200K)

Did anyone try an analysis of these 2 models on various sequence lengths (<4K, <8K, <16K, etc.)?

  • No-Link-2778B
    link
    fedilink
    English
    arrow-up
    1
    ·
    1 year ago

    I have trained book3 for 1 day on a number of GPUs on the 200k, 34B & 6B, it is totally garbage.
    It is not a BASE model at ALL. It even knows itself as GPT sometimes. It was a SFT model on format of benchmarks.
    Try it before you do silly things, you would not find it on SFT immediately, but sooner or later.

    • dogesatorB
      link
      fedilink
      English
      arrow-up
      1
      ·
      1 year ago

      It referring to itself as a GPT could just be from pre-training internet data if it was trained on internet data from 2023.

      • BlueMetaMindB
        link
        fedilink
        English
        arrow-up
        1
        ·
        1 year ago

        It sounds rather like it trained on chatGPT output and they didn’t curate it enough to delete those “As a large language model trained by openAI…” category statements.

        It’s kinda like Shutterstock watermarks showing up in image generation.

        • dogesatorB
          link
          fedilink
          English
          arrow-up
          1
          ·
          1 year ago

          Yea I’m saying that ChatGPT outputs are contained on internet posts in the year 2023, so simply training from 2023 internet data would end up with training on ChatGPT data as a side effect.

          • BlueMetaMindB
            link
            fedilink
            English
            arrow-up
            1
            ·
            1 year ago

            Yes, I understood you. My claim differs in that I think they DIRECTLY used a lot of GPT4 output through the api, which is very probable because a lot of LLM training is done that way. You ask GPT4 to generate examples of conversations with properties you want your LLM to learn and then train on that.

            In order for self identification, as GPT I don’t think that randomly crawled chat Examples from the Internet would be enough.

            I am not trying to make a strong claim on that, it’s just a thought. My people both.

  • DataLearnerAIB
    link
    fedilink
    English
    arrow-up
    1
    ·
    1 year ago

    In most scenarios, models with extended context are optimized for long sequences. If the sequence is not very long, it is often recommended to use a regular model

    • mcmoose1900
      cake
      B
      link
      fedilink
      English
      arrow-up
      1
      ·
      1 year ago

      I felt this too. It seems to “grab on” when you give it a longer context to continue though.

  • mcmoose1900
    cake
    B
    link
    fedilink
    English
    arrow-up
    1
    ·
    1 year ago

    Random update on this, I did some more experimenting on the start of a story (with LimaRP and Petrol LoRAs), and the 4K model seems… fine? So does the 200K.

    I don’t how know to stretch out the base model. Their page claims it supports 32K, but it has a 4K context in the config and no RoPE scaling section. Just a high rope theta.

    The one difference I did notice is that the 200K model really likes to summarize and reference previous parts of the story. Maybe it was trained on retrieval or summarization examples.

  • mcmoose1900
    cake
    B
    link
    fedilink
    English
    arrow-up
    1
    ·
    1 year ago

    I am running my story on 200K, feels the same as 4K to me (which I tried in the same setting before 200K was released).

    And honestly… Even if it is much worse (and I dont think it worse at all), the mega context is such a boon for storytelling.

    What I did not try was 4K stretched out with RoPE alpha or anything like that, but the 200K model does not need any stretching out to at least 42K.