Before you do anything else, I'd make sure you have rock solid evidence before proceeding. I'd ring the number and say, 'Hello, I'm just ringing because I found your number written on a pad in my house and wondered if it was anything important.' Don't say you looked in his phone - just let them both wonder. The reaction you get will tell you everything you need to know. Personally, I'd then pre-empt his leaving by having the locks changed to the house you share and leave his bags outside. Then go and see a solicitor.
I know this sounds harsh but imo you need to take back some control in the situation.
Patricia Greene who played Jill Archer has died

I don't think you are being unreasonably suspicious, I think you are trying very hard not to see the truth. 