The vegan lifestyle operates on the basis of doing the least harm, lightest footprint, etc., and veganism fills the bill very aptly. A vegan lifestyle includes not just the food that's consumed, but also the clothing that's worn. It includes animals as entertainment, it considers the animals that are abused for the sake of the latest body wash, etc. Vegans also acknowledge that there are always going to be collateral animal deaths in harvesting plant foods for consumption, but they are not deliberate deaths as are the 10 billion animals that are raised and killed in the US every year. So on top of the deliberate deaths of all those food animals, you can add in the collateral deaths that result from the harvesting of the millions of acres needed to raise food for all those animals.
Studies have also been done that show a vegan diet to be more sustainable environmentally than a meat inclusive diet.
"...Both the meat-based average American diet and the lactoovovegetarian diet require significant quantities of nonrenewable fossil energy to produce. Thus, both food systems are not sustainable in the long term based on heavy fossil energy requirements. However, the meat-based diet requires more energy, land, and water resources than the lactoovovegetarian diet. In this limited sense, the lactoovovegetarian diet is more sustainable than the average American meat-based diet."http://ajcn.nutrition.org/content/78/3/660S.long
The approximate total number of acres used to produce animals and their food in America is around 1200 million acres. Compare that to about 3 million acres used to produce plants for direct human consumption. Range and pasture land 788 million acres, 227 million acres to grow corn and soy to feed animals, forest/grazing 245 million acres.
http://www.westernwatersheds.org/watmess/watmess_2002/2002html_summer/article6.htm
So no matter how you look at it, the vegan diet causes less harm for both plants and animals ,even philosophically than a meat inclusive one. If the concern truly was for the 'poor plants', then at the very least, the meat inclusive diet is definitely worse as there are 1200,000,000 acres of plants suffering exclusively for the sake of a meat inclusive diet.