Jaggu, might I recommend you also read The China Study by T Colin Campbell. I challenge anyone to read it thoroughly (as well as the resources that Jeff cited as well as the McDougall books) and not understand what is wrong with casein. Whether you choose to eliminate it from your diet entirely is up to you, but you'll at least be making an educated decision.
After all my reading, I came to the conclusion a long time ago that the fat and the cholesterol were among the least of the problems with milk!
I don't have nice references at my fingertips now, but remember that milk is not just straightforward nutrition (i.e. simple calories) for babies (animal and human), it's also a cocktail of hormones and other bioactive chemicals designed to stimulate physical and mental growth appropriate to the species - as well as addictive substances to ensure the babies crave and thus keep drinking it. These are inherent properties in milk even before you get into the artificially added hormones and pharmaceutical agents and so on that are endemic in the dairy industry.
Bingo. I started with Esselstyn's book, but it was Campbell's that really drove it home.
I do not really miss the meat. Feeling the finger on your "off switch" really puts it in perspective.
I do miss the cheese, tho.