Yes... They come by to eat and then leave. I also have rescues of my own (they are old now)...just can't turn away a hungry critter....I plan to buy some more houses this summer (I have 3 now).
I presently have a young cat with five kittens living in my garage. I was feeding the mother cat because she's so small, I didn't know she had kittens.
The Daughters of Discord (neighbors nearby) informed me she had two kittens, then she showed up a week or so later with five kittens. My heart is too soft to do nothing.
I know I'm just enabling more kittens and I know all five probably won't survive, but I'll do what I can for now.