I completely agree that you should do what suits you - if you own your home it is of course an asset, but primarily your home. Especially in later years it should work for what suits you.
You have one spare room for visitors or times when one of you needs extra space. Family can always bunk down in the living room as well, or put up a tent in the garden.
I don't always buy the 'not such a good investment' tale anyway. I have been told this over the years about various changes we have made to houses that suit us, yet I've never had an estate agent say 'oh, we'll have to lower the price because you did this or that'. Bungalows are often bought by the elderly or disabled, and the set-up you propose sounds very good.
Also - does whoever told you this have figure in mind? How much less would you get? Even if £10k, sounds worth it for what suits you.