Another very important thing to consider is the effect he could have on the rest of your family if you got more involved with him or moved in together. I think there is a very good chance he would try to alienate you from your family, including your grandchildren, and make you spend less time with them.
If you are still feeling confused/emotional about the situation, write down the pros and cons of being with him (maybe get a close friend to help) - it does look as though the reasons to leave him far outweigh the reasons to stay in this relationship.
I have experience of both sulking and of manipulation in previous relationships - it's not worth putting up with this, especially as it's not been a long term relationship and you don't have any children together, for example.
You're still young and, as they say, there are plenty more fish in the sea. I got out of a manipulative relationship and, at 48, went on to marry a fantastic, caring man who loves my kids and grandkids to bits.
Good luck.