There is no "one size fits all" state. The factors that bobcat just listed are important ones, and there are additional ones. For many people, being near loved ones (family, old friends) is very important. There are sometimes religious considerations. Just being fond of a place because you have lived there previously and are familiar with it could matter,
I've also found that the things that do matter can change as you go through life. For instance, being able to enjoy a sport might matter when you are younger or middle aged, but be very difficult (or impossible) when you get older. The kind of housing available is very important, of course, and you could be fond of a particular type of house but find that it is scarce in that region. Availability of health services becomes more and more important. For instance, I once thought I wanted to move to Oregon, either the lovely town of Ashland or somewhere along the Oregon coast, which we both loved. But when we saw how far it was to the nearest hospital or airport, we changed our mind.
Some people love cities and all their amenities; others like small towns. Being near a large body of water might matter, or it might not at all.
You mention the affordability problem, which could either be a deal breaker, or maybe just something to grumble about but you can manage to live there anyway.
Sometimes you have to weigh several of these considerations against each other. Example: Vermont is a beautiful place to live for about half the year, but could I stand the winters?
Interesting subject.