For whatever it's worth, I live on the 5th floor, and have been happy here for over 10 years. I have a container garden on my enclosed porch, a little room which is kind of an extension of the living room, with a sliding glass door in between. My flowers are lovely, but I found that I couldn't grow vegetables in pots out there, because the porch is screened, and no insects were coming in to pollinate. I think the lack of insects is a much bigger plus than the lack of home-grown vegetables.
But there certainly is no lack of people. Any time I feel the need for company, I hop on the elevator and go down to the lobby or the patio in back. There's always someone to talk with. This is a very friendly building.
About the elevators, we have three. So there are usually 3 in service, definitely 2. In the event of a fire or other emergency, we would have to walk down, the same as in any other building with elevators. If you can't walk down from the third floor, the emergency workers (firemen, EMT's) would help you. Same is true of a medical emergency. But those events are fortunately so rare, or nonexistent, that I don't plan where I'm going to live around them. I'm more interested in the everyday pluses and minuses.
Does the new place feel homey and cheerful? Is it light enough, at least for part of the day? Is it reasonably quiet? (I once knew someone who bought a beautiful condo which turned out to be next door to a cafe which was popular with the biker crowd. Every night she had motorcycles roaring about 1 AM when the place closed!)
From what you've described, it sounds like moving would be an improvement in your quality of life.