... What, besides the feed, would attract the deer enough to jump the fence? Does the problem lessen, during hunting season? (Or would the barn be a safe sanctuary?)
Adult deer pay absolutely no attention to fences. They go over a 4 foot fence like it wasn't even there---even from a complete standstill. It's rather beautiful to watch them clear a fence so effortlessly. Just the little ones have a tough time at first. Mamma probably knows when they're ready.
My property is a safe haven for deer because I've got it posted. (Didn't you know deer can read signs?) There would probably be more deer during hunting season because they would be chased there from other property. And it has water. It's possible all the deer in the county spend the night at my place, and only go out for extra food in the day. I've never checked at night.