I hardly think it's controversial to state that when humans occupy or make us of land for whatever reason, they displace other animals, indeed entire ecosystems, in the process. They may try to move into these areas once they are established, but may well be considered disruptive or noxious to the human inhabitants and could be subject to pest control measures. Since there is limited surface area on this planet, this means that when humans (re-)make use of land, animals will die for want of living space. This happens whatever the diet of the human population concerned.
When humans do that kind of thing to each other, we tend to call it things like ethnic cleansing and genocide. Viewed in that light, eating meat and wearing leather are merely the gore-cherries on a planet-sized scabcake covered in blood icing. It's not all bad for non-humans I guess though; despite endless attempts at elimination by humans, rats have enjoyed an unprecedented success in breeding and surviving, hitching rides on human transportation to inhabit the entire world, or close enough.
But for the vast majority of non-human animals on this planet, the mere presence of humans is
usually bad news. What is more fundamental than the right to life? How can animals be liberated under the shadow of human domination?
The problem as I see it is that being vegan is not even close to "100% watertight", if one takes the view that killing animals is inherently wrong. Would you accept a war criminal's excuse that at least he and none of his underlings ended up eating the bodies of their victims?