I am in exact the same situation as you. There would be no problem at all if your wife stay in Canada.
. It should influence the time you get the citizenship. The custom officer in the airport could not refuse you to get inside Canada cause your wife is there, how could they seperate a family?