I don't think the so called soft look is from fat gains, but its more from what in the milk. With bodybuilding things are complicated or from what I know they are, different people respond to different foods different (this is why stepping on stage is so hard, because you really need to know your body) some people will hold water from drinking milk, things like that. Also milk can be a little bit fatty but if you make it a staple of your diet and your drinking tons of it, it can probably put on a few lbs. I believe the soft look from drinking dairy can come from the sugar (but dont quote me on it)

now that your read my rant in conclusion I don't see the need to worry about it to much unless u plan on steppin on stage or something