You're absolutely right. I make assumptions about people all the time. It's one of my favorite things to do.Yes you are free to dislike them, think they are ugly etc
Seeing them as a sign of low class, bad breeding etc is different though - that goes past your own preference to making assumptions about other people... aka prejudice.