I think your characterisation is a gross oversimplification, and one that's quite dismissive of the fears of some particularly vulnerable people.
Not only does your framing of the issue fail to take into account psychological harm that arises from fear (which occurs regardless of how 'legitimate' that fear is), but, also, it plays fast and loose with the idea of threat. Women are well aware that they live under constant threat from people with penises (the only people in law who can commit rape); as a group, they represent a threat - that's the basis of single-sex protections for women. It's quite possible to argue that some individuals or sub-groups within that group represent less risk, but there might be good policy reasons not to favour such exceptionalism. For instance, any individual could argue that, absent any specific evidence that he is more likely to harm someone than a woman who is allowed into a space, he should be let in. But I don't need to explain why that's resisted.
Now, as it happens, I agree with you that the empirical evidence from places which have allowed inclusion suggests that it does not result in more attacks on the women who would already be in those spaces (and reduces harm to women who might otherwise be excluded). And so I'd like to see a world where women are keen to welcome trans women into their spaces (albeit I'm less keen on that being imposed on those who aren't). But I don't think that necessarily means all counter-arguments are analogous to racism.