I’ve been closely following the recent developments from NVIDIA, and their latest announcement has really caught my attention: the H200 with the new GH200 chip. This beast is said to pack a staggering 141 GB of RAM and offers a blazing 4.8 TB/s speed. The premiere of the H200 is slated for the second quarter of 2024, and I can’t help but ponder its potential impact.

The most exciting aspect for me, and probably for many of you, is its capability to run LLAMA2 70B at twice the speed of the current H100. That’s a significant leap in performance!

So here’s the big question for the community: are any of you planning to upgrade to the H200, or are you planning to stick with the H100 for a while longer?

I’m currently using the 8xH100 rig and it’s been a workhorse, but the prospect of doubling my LLAMA2 70B performance is very tempting. However, I’m also weighing the cost versus the benefits. The H200 seems like a substantial investment, and I’m wondering if the performance gain justifies the upgrade, especially considering the still-capable H100.

I’d love to hear your thoughts, experiences, and plans.

  • a_beautiful_rhindBEnglish
    1·
    10 months ago

    You’re doing this all wrong. You wait till someone buys the H200 for you and their startup fails. Then you scoop it up on the low during the fire sale to appease their creditors.

  • AaaaaaaaaeeeeeBEnglish
    1·
    10 months ago

    The most exciting aspect for me, and probably for many of you, is its capability to run LLAMA2 70B at twice the speed of the current H100. That’s a significant leap in performance!

    You didn’t read the github page (show)notes, so I think you are being misled.

    • Its not going to give 2x inference speed (if you are renting this for inference), it is going to give up to 1.4x.

    • 1.4X bandwidth compared to h100 “H200 is the first GPU with HBM3e memory with 4.8TB/s of memory bandwidth, a 1.4X increase over H100”

    • 1.9x throughput on certain sequences.

  • Herr_DrosselmeyerBEnglish
    1·
    10 months ago

    I’m currently using the 8xH100 rig.

    That would be a DGX H100 then so I’m assuming you’re using it in the context of a business so you’ll need to crunch the numbers, we can’t do that for you.

  • laterralBEnglish
    1·
    10 months ago

    Honest question, what do you use this for? Sounds like you have immense local firepower!!

    • Wrong_User_LoggedOPBEnglish
      1·
      10 months ago

      that was only my dream, in the morning I woke up and sit back to my 16GB of VRAM… 😅