Personally I always thought that the Viltrumite’s superiority complex was a factor here. They view the Coalition as an annoyance, not a threat. I mean Allen, the Coalition’s best fighter couldn’t beat a single Viltrumite soldier in combat.

Also taking out the Coalition would be a logistical nightmare. The Coalition isn’t one planet it’s dozens of planets. To deal with them, the Viltrumites would need to conquer all of the planets that make up the Coalition and they can’t because they’re just stretched too thin right now. It’s why Nolan was sent to conquer Earth alone, he and his fellow officers were each assigned a planet to conquer and given a timeframe of 500 years to do so (that’s the reason he specified that amount of time to Mark during their fight). The Empire just doesn’t have the resources to mount a full offensive against the coalition because of the sheer size of it, so they leave the Coalition alone for the most part, let them gather worlds and species to think they have a chance against Viltrum, and then when the time is right, take out the Coalition government, squash any resistance and in one fell swoop add the entirety of the Coalition to the Viltrum Empire.