"is there anything about me that annoys you?"

Yes.

I don't like that you won't let me be nice to you because of your gentlemanlyness.  You started taking the piss instead.

I don't like that you took the piss out of me when I wanted to take your lunch into you.

I don't like that you took the piss when I generally try to be nice.

Apart from that.

You're great.