Update on the post immediately below:
My friend D--who happens to be a kung-fu expert--has convinced me that racial slurs are not the same as physical attacks. His reasoning is that they are psychological attacks more than anything. If the violence never actually begins, one usually walks away with nothing more than anger and stress.
Racial slurs do indicate intent for physical harm though, and I think, given the context of their historical use, that they are indeed physical threats. With a racial slur, the speaker indicates the intent to cause imminent physical harm. Calling someone a "chink" in my opinion is akin to saying "I'm going to beat the @#$ out of you." In my opinion, if one fears the threat, one has the right to physical self-defense. Of course one wouldn't be justified in getting physical against an 80 year old woman who shouts a racial slur (unless of course, she had a gun...I think that would be more comedic than anything else), but in my opinion, an angry young man (or men) who uses such a slur has indicated an immediate threat to cause physical harm.
And in such cases, I believe it is morally justified (and maybe morally imperative) for a person to defend himself.