I love dairy but avoid it, for the most part
. I have not noticed anything personally bad from it, but I will run with the general consensus on this one.
However, I did try adding ISO Whey Protein to my diet (mixed in Soy Milk) in the hope of building my strength. This in my diet did cause me issues, and I found I did better without it.
I think different people will/do react to things differently.