Your summary question and answer section, "What else is going on?" is what I was looking for.

I posted something about a similar situation here on medium a couple of years ago. What happens if Damon's wife comes to you in confidence and says, "I think Damon may be cheating on me. What do you think?"

My personal experience as well as a lot of studies have shown that who someone's close friends are says a lot about them. If they have close friends who cheat on their wives, what other kind of values do they have? If your husband goes with them on a "guy's weekend" you have to logically think, is it really just guys?

This line of questioning can go on for a while, and leads nowhere good if you want an exclusive relationship with your significant other.