A lady I know lived as an unmarried trio with her two sisters for many years. She got on well with them, but has had to cope with first one, then the other suffering ill-health and dementia, before they finally died. She now lives alone in a big house, but is enjoying the freedom.
Another neighbour moved in with her sister while some work was done on her house. Both widows, they enjoyed the arrangement so much that they stayed together, spending the proceeds on having fun, when the other house was sold.
I think that I would try a 'trial run' with your brother, before making a definite commitment, and ensure that you both have somewhere to be alone if the mood takes you, too. It sounds like a good plan, but things don't always work out as you expect. Good luck!