That’s a heavy thing to have down the road, especially knowing that those groups aren’t just about free speech but often about intimidation, exclusion, and potential violence. If a Neo-Nazi compound opened near me, I wouldn't ignore it. I think silence or passivity is part of how these groups gain ground in communities.
First, I would try to find out what the local and state laws say. Sometimes zoning laws, permits, or code enforcement can be relevant, even if not directly tied to ideology. You can’t ban a group for hateful views alone due to free speech protections, but you can hold them accountable for breaking any law, even a minor one.
Second, I would connect with neighbors, local clergy, community groups, and possibly civil rights organizations to create a united front. The goal wouldn’t be violence or harassment, but visibility, pressure, and civic unity. Hate thrives in isolation. A neighborhood that is informed, organized, and vocal can make it clear that hate isn’t welcome.
It's also worth contacting local law enforcement. Not to call for a crackdown without cause, but to express concerns and ensure they are aware of any potential risks. If history has taught us anything, it's that these groups can become dangerous quickly.
What I wouldn't do is shrug and hope they go away. Too many communities have done that and regretted it later. All that said, no way I'd confront them as an individual nor draw their attention as an individual. Nope.