Definitely not the docs. I don't worry about the number too much either. After working out and trying to keep up with fitness news for decades I worry more about physical symptoms than numbers. I know my body pretty good and my observations are much more accurate than in years past. One has to be honest with themselves.
The biggest I worry about is how out of breath I can get. If I get fired up for what ever reason does it bother my heart, am I getting headaches, do I feel to hot or cold, do I fee tight. I also pay attention to hydration, if I feel thirsty for any length of time I not only increase water/fluid intake but begin to wonder if do I have diabetes or are my kidneys cooking.
One needs to be acutely aware of what's going on in/with their body. And/or how your body responds to what you do it wether it's food, activity or environment.