You can be as critical of them as you like. It's no skin off my nose.
If he doesn't come to my party it's no skin off my nose.