How did you know about the "rumors"?  If you hear someone talking about it, you simply clarify it to those people.

On the other hand, be positive and open to your co-workers.  Chat with them more, take part in their activities more.  Rumor will not stay such people.