For me, it's not about religion (although I do resent proselytizing). The problem is intrusion into my private residence and my private space when I'm just trying to live my life and do whatever business I need/want to do.
I don't care if it's Jehovah's Witnesses, Buddhists, or Save the Whales.
I remember an incident when I was eating at a diner and someone came up to me and asked if I had accepted Christ as my savior. When I politely asked the person to leave me alone, they said "Hmm, well I guess you're uncomfortable when someone tries to save your soul." Well, they can guess whatever they want. They don't know anything about my comfort or my soul....it's not their business.
As I said before, everyone has the right to be left alone. I don't have to answer your questions, listen to your self-righteous preaching, allow you to toss the Bible in my face, debate you, or even listen to you.