Eventually, every city dweller has to learn to say, "Fuck you man, you don't know me. Why you want to start some shit?", possibly while pointing to a nearby cop.
Translation: "I say old boy, I'm an unknown quantity that is mostly likely not worth an indeterminate amount of risk. Why don't we go on politely ignoring each other?"
It sucks that you end up having to do this, but it's the only thing I've found that reliably works.