That may be true, although it's already a welcome step back from your earlier claim. If we can get away from the cartoon politics though the reality is that the uncouth racism represented within Leave and the sanitised, managerial racism represented among, for example, 'moderate Tories', Lib Dems and the Labour Party (and for that matter the EU itself) of Remain, work hand in hand.
If all racists voted to leave then many of the architects of racist policies who sustained, enouraged and attempted to make political use of that racism supported remain. For purposes of clarity I don't think that makes everyone who wanted to stay in the EU in some way morally compromised.