Just tell him you only want to have a professinal relationship with him. Alway keep the converstion in business. Don't talk about anything else. Be polite but not friendly. then if he do anything cross the line, you can let your manager know.