Maybe it was the Spongebob Squarepants speedo you were wearing?
Seriously, physical degradation is a fact of life. But you don't have to go gently into that dark night. I work out 5-6 days a week so that I don't slide back into "dad bod" territory where I was headed ten years ago. My wife is 11 years younger than me, so that keeps me young too.
But mostly it is my attitude. If you weren't going there to be with someone except your wife, who gives a fuck if they leave? More privacy for the both of you.
If something like that happens agian yell after them, "Don't think I didn't see those extra bubbles you made!" I guarantee a serious laugh for you as at least one will stop and be confused about if she should try to deny it or if saying something would make it worse. Priceless.