My best understanding is that most of the cholesterol in your body is cholesterol that your body makes. Diet is a relatively small factor (though it can still be used to mitigate high cholesterol to some degree). This is why medication is more effective for lowering cholesterol as well, because it impacts upon the larger source of cholesterol in your body, not just what you ingest.