I did realise that - wasn't confusing the yellow in the EU flag with LibDem yellow!
The thing is, I think you're conflating a group of pretty fanatical remainers (said flag-wavers/FBPEs) with an enormous section of society, just under half if you use the research industry standard ferrelhadley quoted, a lot more than that if you use the Marxist definition of not owning the means of production. The whole situation is a lot more nuanced than that.
A lot of working class people will have voted to remain. If you add the issue of BAME voters into the mix, a majority of them voted to remain.
This is from Parliament's own analysis (it actually surprised me as I would have thought that there would have been more of a link between poor socio-economic conditions and voting to leave, simply because of the - understandable - desire for change and the perception of a vote to leave as a protest vote against the status quo):
There is no obvious relationship between the proportion of people claiming Jobseeker’s Allowance, and the proportion of people voting Leave in a local authority. Likewise, there is no obvious relationship between the proportion of children living in low income households in a local authority, and the proportion of people voting Leave.
This suggests that votes for Leave and Remain are not strongly related to this set of socio-economic indicators individually.