Perhaps it wasn't them personally you didn't like, just the idea of your dad's workmates was something you didn't like; him spending time with them instead of you?