"Not seeing colour" has always struck me as one of the strangest claims that people make about racially-related issues.
To say "colour is not the prime criterion on which I judge someone" is right and fine, of course.
But to say that "their colour does not in any way affect my perception of them" is not only either a lie or a self-deception - it's also, as I think you imply, a part of the problem. Colour may have been a fundamental influence on a person's life; refusing to see it is not a virtue but a kind of self-blinding.
(When I say it's "not a virtue" I don't mean it's evil, either. I am sure it is often meant with the best of intentions.)
If we literally "don't see colour" in certain societies then we don't see why there is such a problem, and we don't fully see the person we're talking to. Much better, surely, to frankly acknowledge that we *do* see colour - and just try not to draw unjustified conclusions from it.
(As a postscript: on a literal level "not seeing colour" is a frankly absurd claim, though some people extend it even that far. We see height, age, body shape, many other things at a moment's glance, and colour is no exception.)