(1) You don't feel good and right if you just let go. therefore, can't let him come. NOT NOW.
(2) You don't want to completely give up the marriage, maybe just for the sake of your child. therefore, you won't sign the devorce paper.
(3) Imagine in 1 year or 2, what do you want? You forgive him and he joins you two here. You both forget the past and look forward. If that's really what you want, give yourself an excuse to forgive him, such as have one or two night stand, just so that you get even. (don't get mad, get even).
(4) I bevlieve point 3 is very flammable ... I'm waiting... :)