To put it simply...

I don't care if they're rich. In fact I'm glad for them. It must be nice, for reals.

What does rub me the wrong way is when one of them tries to convince me they're not rich.

That makes it look like they think I'm stupid.

I do care about that.