If someone goes off with another person, they leave their husband, wife, or lover and have a relationship with that person. I suppose Carolyn went off with some man she'd fallen in love with
If someone goes off with something that belongs to another person, they leave and take it with them. He's gone off with my passport