The Labour Party is responsible for legislating racist laws, for implementing and pushing racist policies, creating a racist climate from e.g. distributing racist anti-immigration mugs. It is a political group that organises for power and to use that power for the ends it supports. And like any political group must be evaluated on the basis of its actions and its politics.
ETA: To expand - the LP, like the Conservative Party, the SWP, the ACG or whatever else, is not just a group of people with shared interests, or who just happen to be thrown together. Even less is it comparable to society as a whole - a vast complex body which one does not volunteer and pay subs to join.
No doubt the Royal Society of Chemistry, the RSPCA, the National Trust, CAMRA, The Friends of Hyde Park Picture House and the West Leeds Donkey Lover and Debating Society all reflect the systemic racism that is present in society. And while no organisation can be totally apolitical, none of those organisations (even those arguing for policy in particular areas) are organising on the basis of forming a government. To put the LP in the space basket as the RSC, let alone society as a whole seems a strange thing to do.