Hello everyone,

I have an issue with my desktop, which I really don’t understand.

It is a desktop PC, with a Nvidia GPU, which I use for development (of AI, but like the image analysis part, not the slop part). I usually let it always run, and my previous machine was working fine like this.

However, on this machine, I get crashes after either a few days or a few weeks.

These “crashes” are weird : the machine is still up (and I can ping it), however I can’t ssh into it, I can’t connect to cockpit either, and can’t login physically on it (the display is frozen).

The only way to get it back up is to forcefully shut it down, wait a little and then powr it back up; but this isn’t ideal, since it interrupts computations etc

The only crashes shown in journalctl are related to irrelevant (I believe) programs : KDEConnect, DrKronqi, etc; nothing right before the crash.

Do you have any idea what could cause this ?

I tried logging temperatures, but nothing abnormal showed up. I ran memtest86+, nothing showed up either. Smart tells me everything is alright for my drives as well.

Right now I’m thinking about changing the alimentation or the motherboard, but frankly I don’t know what’s going on and it’s really puzzling me.

  • brucethemoose@lemmy.world
    link
    fedilink
    arrow-up
    4
    ·
    3 days ago

    Get your AI tool to write its output to a log on disk.

    Get another program to log ram and vram usage periodically as well.

    As my top suspicion is your application has a memory leak, like many such applications do, and it just gets the system to hang when it runs too long.

    There are ways to fix this too, like PyTorch allocation variables, inserting some manual garbage collection, or just periodically restarting it.