My MIL had severe back and leg problems; she had a handicap tag and needed it. But when she first got out of the car you wouldn't have been able to tell right away. She would be in awful pain after going to the grocery store; the permit parking was a necessity for her. People would accost her because their eyes told them she didn't qualify for the permit. Fortunately her doctor knew better. Later in life she used a walker and the harassment pretty much stopped - but before that people's visual expertise would really get her down.
Exactly this.
Plus I'm probably only good for a punch or two before I can't breathe, my heart rate gets too high, and I get zapped by my defibrillator.