I've been asking myself the same questions, lily. I agree with what holly says: if the segregation is to supply privacy for intimate moments, then it is acceptable. Really, if nudity or sexuality is the issue, then providing privacy applies to everyone whatever their gender. Some hospitals have mixed wards. So long as privacy is maintained where needed, that shouldn't be a problem.
I agree about the gym or swimming sessions too. I don't think there should be one-gender sessions anywhere public.
I suppose private clubs can do what they like but the Royal and Ancient Golf Club at St Andrews has recently opened its doors to female members, which is good and about time.