It really depends on what stage of society you're talking about.
Capitalism is prone to creating the conditions and mindsets for monstrous behaviour, and this in turn creates the conditions for mass incarceration in conjunction with the methodology of a centralised State. As posted by various people above, there are certainly examples of best practice to try and maximise just outcomes even within this situation, and there's fairly extensive evidence that properly-managed social safety nets enormously reduce the need for prisons (while badly-run or neglected ones end up with the "just one more prison bro" approach, and the revolving doors of the university of crime).
If talking about the "post-revolutionary" model though it becomes much more difficult to say because we have no real idea how much of the abusive behaviour we see today is nurture, and now much nature/happenstance which can't be stopped. Would a post-patriarchal society in which constantly battling for primacy as a Social Good is no longer fetishised eliminate a bit, some, most or all of the paths down which people fall?
I think very few people would say "all" but quite a lot of socialists might say "most".
In that latter case, then, you do still have a very small number of people who are a continued danger to others for whatever reason. But how does protecting everyone else work? I'm not a fan of the exile method, personally, because it simply ends up with abusive people either being isolated to the point where it becomes nearly as cruel as prison, or they are unmonitored and free to simply show up elsewhere and damage more lives. I do not, however, think that establishing central facilities is a good idea, either for them or for the
people guarding them.
Kropotkin's solutions involved two possibilities. One, that you find ways to semi-integrate such people in the community where you can while keeping them in hand, aiming to re-establish their connection to society and retrain them to abide within it (eg. you warn the community about them, make sure they're kept an eye on, but also allow them to eg. work outdoors growing things, while interacting as constructively as possibly with others, likely volunteers who are good at/trained in that sort of thing). Two, that you isolate but don't guard them (he claimed he'd seen a colony of exiled murderers in Siberia who, freed from the desperation of Moscow and St Petersburg, turned their lives around completely). I think the first is an option, the second a perhaps overoptimistic position but maybe depending on circumstance. And both could potentially fall short where it comes to the worst of the worst. But I do think he has the right idea when it comes to not banging damaged people up en masse behind big walls where they can be forgotten about by most everyone else and have no opportunity to change at all. And in a world where we've eliminated most of the problem through changes to social norms, Kropotkin's basic vision of intensive, holistic consideration for each case becomes much more realistic to do.
We tend to forget, in society as it stands, that the prison system is a direct reflection of our priorities as a society - dump the troublemakers somewhere, as cheaply as possible and in a way that serves as a threat to the rest, for the sake of a smoother productive process. It's an industrial solution to what's actually a human problem. The existence of "irredeemable" humans is used, systemically, as the justification for an abdication of humanity.