There's no simple answer. We used to segregate hospital wards but no longer do in many cases. Other countries have less segregated facilities with no ill effect (especially Scandinavia iirc).
Prisons are a thorny issue, because a victim cannot escape their aggressor, and because the system is even worse at protecting them than it is outside... yet this is a problem between cis men in prison - and in this country, apparently a very serious issue of significant magnitude between cis women in women's prisons, sexual violence being used to enforce prisoner hierarchies.
Another thorny issue is women's refuges. Yet already these do not offer refuge to the mothers of sons above their individually determined cut off age.
Organising society along sex-binary lines is not a clear or unproblematic thing to do. The introduction of trans people to the equation, and of gender-non-conforming people complicates matters, but saying "fit in with the existing system, using your genitals OR genetics OR legal status OR self determination" (depending on the preferences of the speaker), doesn't solve the problem for anyone except the speaker.
Even for gender conforming cis people, our systems are like trying to put a square peg into a round hole. When we realise that our society isn't binary at all, we have something more asking to a great big spikey star-shaped peg and the same round hole.
The answer isn't, and has never been, "hit the peg harder" - because the people at the edges get broken that way. The answer is to change the shape of the hole.