"If I wear a an Alpaca ear flap hat from Peru, I am just wearing a hat. I am not pretending I am Peruvian or attempting to make fun of Peruvians — I am wearing a colorful and warm hat."

Exactly this!

IF the hat has some very strong, deeply-rooted and widely respected cultural connotations - for example, if it is a hat only supposed to be worn at certain times of year, or by a certain group in society - then we get into a more sensitive area, for sure. Your example of "The Roman Catholics" illustrates this well.

But most of the time, most things don't.