Its pretty bad when the public is not happy with the PM. I've heard many complaints through the years of various PM's/OIC's but not to the extent that you speak of.
Understandably not a popularity contest but in small towns its not hard to be appreciated if the alternative is someone that is cold in nature. So fortunate for me, I get alot of happy customers that tell me how much they appreciate my work. Sad thing is I'm not doing anything special, it just looks great in comparison.