to bother, disturb, irritate, or annoy

listen to the pronunciation of to bother, disturb, irritate, or annoy
English - English
rub somebody the wrong way

It's a small thing, really, to leave towels unfolded, but it rubs him the wrong way.