A 50cc small bike is heavier than a pushbike of course, but realistically you're going to be similary hurt by a bicyclist running into you at 20 mph as you would by a 50 cc scooterist at the same speed, even if the former has a combined mass of 95 kg and the latter 140 kg. They are broadly similar. If being hit above 20 mph by a mass of 140 kg is an unnacceptable risk, then sure as fuck so it is being hit by a 90 kg mass.
But even if you still objected to that, surely you would still have to admit that ebikes should be subject to such speed limits, becaue they are certainly a lot heavier than pushbikes, and a fat bloke on an ebike will probably weight just about as much as a small person on a silly 50cc scooter.