The whole "collective responsibility" schtick is a convenient political fiction that displaces blame onto an entire polity.
However, as the majority of British people had no voice during the majority of empire (most men not until 1832 at earliest, women only in the last 4 decades that the institution existed), they had no leverage on the power exerted on the dominions by the ruling class. The only "collective responsibility" lies with that ruling class, and on those still "sitting pretty" on fortunes amassed through robbery and rapine abroad.