• Pennomi@lemmy.world
    link
    fedilink
    English
    arrow-up
    3
    ·
    21 days ago

    A lot of the smaller LLMs don’t require GPU at all - they run just fine on a normal consumer CPU.

      • Pennomi@lemmy.world
        link
        fedilink
        English
        arrow-up
        3
        ·
        21 days ago

        It depends. A lot of LLMs are memory-constrained. If you’re constantly thrashing the GPU memory it can be both slower and less efficient.

    • DavidGarcia@feddit.nl
      link
      fedilink
      English
      arrow-up
      1
      ·
      20 days ago

      yeah but 10x slower, at speeds that just don’t work for many use cases. When you compare energy consumption per token, there isn’t much difference.