Of course golliwogs weren't designed as a tool for racism - the concept of racism, as we recognise it, didn't exist at the point that the golliwog, as a depiction of black people, took its rise.
But a whole cultural iconography around golliwogs developed which is, generally speaking, regarded as treating black people as somehow "less than" whites - a stereotype which demeans them and writes them off as childlike, less competent/serious than white people, etc. And, from that, a view that the depiction of golliwogs is associated with what we generally now consider to be racist attitudes towards black people. Perhaps it wasn't meant that way, and perhaps that small number of people whose fond memories of the iconography is so strong as to override the racist connotations are genuine. But the point is that, nowadays, most people DO associate golliwogs with at the very least a patronising attitude towards black people, and anyone who insists, whatever their justification, that that association doesn't exist is going to get a hard time for it.
You could argue the same about the word "nigger". After all, it's only a collection of letters, and it takes its origin from a Latin word meaning "black" - why on earth should anyone regard it as offensive? But the fact is that it is a term that has come into common use as a derogatory word for whites to use about people of colour, and anyone using it otherwise needs to be aware of that baggage: generally speaking, it's just easier to avoid using the term. Tyranny of the majority? Perhaps. PC gawn maaaad? Quite possibly. Live with it.
ETA: better make some things clear.
I don't mean racism didn't exist when golliwogs took their rise - simply that the idea of it as being an exceptionable way of thinking wasn't exactly mainstream.
I DO think that a lot of the depictions of golliwogs were, at the time, "innocent" in that they were simply a reflection of prevailing attitudes, rather than an explicit statement of racism.
And I think that a lot of the equally innocent spluttering and protest by people today who maintain that their own particular use of now-considered-racist iconography and terms is equivalent to that: they may not be actively promoting racism, but in their insistence that this iconography/language should continue to be normalised, they are to some extent legitimising it.