A while back I was developing a workflow scheduler. However the run duration kept coming back negative. The runs kept finishing like 15 seconds before they started. Eventually I figured out why; the company had configured a policy to use their custom NTP server, but it was 16.7 seconds off. So my Windows laptop’s time was in the future, but the WSL VM (which ran the program) used another NTP server and was correct. In the end I just added a bodge that made the program request the time from an external server on startup to calculate an offset.

  • naur@tech.pr0n.pl
    link
    fedilink
    English
    arrow-up
    6
    ·
    5 hours ago

    I once spent several hours debugging a performance issue in the Linux builds of a project. The service worked fine on Windows but couldn’t keep up with the traffic when deployed on Linux. Interestingly, CPU load was actually higher on Windows, and no sampling profiler showed any suspicious hot spots.

    It turned out the problem was in the program’s main loop, which was OS-specific and called something like sleep(float seconds) (yes, I know…). The parameter was set to 1e-3 on Windows and 10e-3 on Linux. I stared at that line several times before I noticed the difference.


    Another time, all services of a distributed system were regularly reporting a loss of communication at 4 AM. The servers were supposed to receive data continuously, so even a few seconds of silence was enough to raise an alert.

    Half of the team investigated all kinds of possible locking scenarios (the applications were multi-threaded). Then we noticed that all the logs in the system (including the kernel’s) had the same gap in timestamps.

    Then someone remembered, that the customer had a cron job which did some maintenance at night. They confirmed that it halted the entire VM with all services for about 5 seconds.