The way in which the majority of people understand racism is this line as applied solely to the act of prejudicial behaviour against someone of a different race (incidentally largely disqualifying "gammon" as it's used primarily by white liberals if Twitter is any guide). Which okay fine, meaning is what people make it, but misses a chunk of situational factors which make quite a big difference to the conversation around its use, importance and impact.
The basic secondary analysis here is that in any society where you are the disempowered minority it is not practicable to enforce prejudicial belief in any sustained way. Yes you can be a prejudiced prick, you can believe what you like, you can even in some circumstances bully people (eg. if you have an individual school with a black majority in a mostly white country) but that doesn't mean you will ever be able to make significant inroads into how the dominant majority is regarded or treated on a day-to-day basis. And it's that power which actually gives racism meaning. I've never been spat at in the street for being white and if I was it'd be such a shocking event that it'd make a page lead if the press picked up on it. It would be a racist event in popular parlance but for me, it would likely never happen again and thus is not part of any sustained social culture or politically-indulged policy that systematically attacks me.
Personally I think the argument that "you can't be racist if you're white" is a bit self-defeating because (like the use of the word "privilege" when applied to male rough sleepers) common use and understanding of the term encourages rejection of the actually quite important core of the critique. But it's not as ludicrous as some people are suggesting here.