You should be polite but very assertive. Chinese people don't like to say No directly, but here it may mislead people. If he is harrassing you, don't back up. Talk to HR