Do you reply the exact same way every time you encounter the same situation/phrase? I sure as hell don't. In fact, I'd even go as far as to say that lizards react the same way to things and that humans have different reactions depending on their mood. Maybe the day you said it to him his boyfriend had just broken up with him. Or, mayhaps he knew "fuck you guy #2" personally and it didn't bother him so much when that guy said it.