I don't worry about it. But the situation here has changed. Our valley, in the '70s-'90s, was Mayberry safe. However, during that time a lot of households started growing cannabis. Some for their personal use only, some as a prepared black-market crop.
Most of the growers of both sorts were gentle, friendly people who simply wanted to afford living. A relative few aspired to be high rollers, and they made connections with biker-gangs that financed their operations & distributed the primo weed. When our federal gov't legalized pot possession & began to regulate production, distribution & sales, the market price for illegally grown pot fell dramatically. I've heard eventually to 30% of the former.
Some of the bigger operations continued to cultivate, because their volume of output still afforded a livable income. And some of the bikers who'd bought property in the area remained. This segment of the populous is a small but conspicuous pissed-off circle... most of them alcoholic, some using meth, and a small number reputedly able to afford coke. Curiously, the incidence of burglaries (mainly of business locations) has increased since pot legalization.
So, while we didn't before, we now do lock doors. But, day or night, I don't feel unsafe when I'm out and around in the community.