Our county prisons are right in town and there never seems to be any problems. I suspect they are quiet as neighbors too since their behavior is strictly controlled. Still though, I would not care to live across the street.
There usually are areas in cities that have been abandoned & can be reclaimed for jails/prisons that aren't near residential areas. Jails with rotating customers should be nearer the courts where they are to appear to make it easier for the authorities who are responsible for the transportation to get them to the hearings.