You're absolutely right, of course - those people deserve solidarity, not anger (except those from the groups above who voted Tory or UKIP - and there would a few).
The problem is, that even amongst Labour voters, there's a large number who would position the party further right than they are, who have no real desire to tackle the fundamental problems faced by society (such as unsustainable economic policy, pandering to multinationals and banks, signing up to things like the TTIP, PPIs, etc. etc.). So if you lump them in with the Tory, UKIP and LibDem voters, you're probably looking at about 75% of the English population who would collectively vote either for the status quo or a move further to the right. I think it's perfectly reasonable to say to them, "you reap what you sow".