That's the real issue. People have got caught up in talking about the slight liberalisation of work visas when the real issue is the end of the UK 2 tier immigration system. One where the UK had one set of immigration rules for predominantly white EEA nations with few restrictions, and a extremely restrictive and draconian system for everyone else. Now all but one of those EEA nations are going to be in the extremely harsh and draconian system. I've heard a lot from the left bemoaning the end of a two tier immigration system. I've heard little about how a extremely restrictive and draconian system can be made less so.
However on the issue of the end of free movement for commonwealth citizens having nothing to do with the EU- I think that's a pretty ahistorical take. The same piece of legislation that put the final nail in the coffin for free movement from commonwealth countries also laid the immigration ground works for the UK joining the EEC
(1). The EU has frequently required new members to close open boarders, regularise migration and bring in harsh restrictions. Similar policies were demanded with Spain closing down migration from North Africa and the EU Commission supervising Poland's restrictive new immigration rules and closure of the board with Ukraine
(2).