Unfortunately you cannot spot target fatloss. Most men store fat in the gut/love handles so your not alone. Really all you can do is make diet and exercise adjustments in order to lower the amount of calories you are taking in or increase the amount you are expending. Take some starting measurements, make a few small adjustments, and then keep track of your progress to see if any other changes need to be made. Eventually you should see some results.