Right. It's really not going to allow the core as a whole to be able to stiffen and strengthen. When going for a max I'd say no big deal of course. The thing is though, you can logically ask how are you going to take the entire body to the next level if you leave any muscles out during the big lifts? Also, you can assume that if you always "test" using a belt but then don't use a belt for the relatively lighter intensities you've actually thrown off the relationship between your maximal ability at any one time and the weights you use around that. So in other words, without the belt you can end up lifting too heavy and risk overworking, injury, etc. even before the long term affects of the belt. Even if the belt doesn't allow you to lift that much heavier it certainly affects things.
So think about this. Do you want to have to lift lighter than normal in relation to your "heavy" lift you did with the belt?
If I'm doing my max work and I lift 375 with the belt for a relative max, I don't want to then have to rely on the belt to do some good work in the 320 to 340 range, for instance OR have to do less work than I normally would or do it lighter. I want to keep it all real.
But to me it can be different when I'm testing or going for a PR versus when I'm maxing on some particular day.