This is a more complex question of course, and there are no easy answers. Some points to make that though are that the number of people in this situation is fairly low, and that they are also at risk from other people who are vaccinated, as the vaccine does not prevent you from either getting the virus or passing it on (although it does of course reduce the risk of infection and hospitalisation). The question then is how we weight one risk against another.
Is the risk to society of changing the relationship between state and individual greater than the risk of allowing people to make their own vaccine decisions? I would argue that if we put in place a precedent where the state is able to override your bodily-autonomy then this opens the door for all sorts of horrific policies in the century to come.
Is the risk of the virus greater than the risk of putting in place technological infrastructure (tracking apps, etc) that could, if the wrong people get in charge, make an authoritarian, survelilance state a real possibility?
How do we weigh these risks? I honstly don't know, but I haven't heard anyone even attempt to.