I would hazard a guess that when a term like "nitty-gritty" is used, more than 99% of those who use it and more than 99% of those who hear it have no concept of its origin (assuming for the sake of argument that the etymology is correct).

Can racist speech (as opposed to action) really exist in the abstract like that, without anyone intending it or perceiving it?

Moreover, if it *were* racist to say "nitty-gritty", why would it not be racist to say "plantation", or for that matter "slave"?

They refer to the institution of American/transatlantic slavery in a much more explicit and widely-understood way - and unless I'm misunderstanding you, referring to the historical fact seems to be the only objection to "nitty-gritty".