It all depends if your muscles are fatigued from work and affect your workout. If your fatigued going into the workout then you're intensity will be way down. If you can recover from working with a good nights sleep, I don't see how it would affect your muscles negatively. Plus if your lifting alot of weight in the gym compared to the freight, it shouldnt affect you.
If anything I would say it might actually help muscle growth.
I really wouldnt worry about it much, unless you're gonna quit your job in the pursuit of lean muscle
