But once I’ve hit my goal and don’t need to hit gym that hard anymore… That frightens me.

I’m pretty sure the notion of not needing to exercise as much after you’ve hit your goal is a misconception to begin with, if it makes you feel any better.