If you allow someone or some people to cause you to decide to do something against your better judgment, you are allowing them to control you. All of us have the power to control certain things in our life and having the power of free speech is one of them, so long as we are respectable. That being said, it’s your choice whether to stay or go, no one else’s.
If you allow those people to chase you away, they win. The do not receive a prize, but they do earn bragging rights. Those people are known to have a personality disorder.