"I was taken aback and didn't say anything but my son told him off mildly. "
Just as well, Imo. If a parent is there, it's really their job to correct their child. (I probably would have been taken aback too.)
And I'm glad ds scolded him, but did it "mildly." I agree with those who say gs needs to learn what's appropriate and what isn't. That's part of a parent's job. But I think it was wise of ds to correct gs mildly since I don't think gs meant any harm.
"I was a bit annoyed at first but my husband and I had a good laugh about it in the car on the way back."
That's the place to do it, imo. Or anywhere totally out of the child's earshot. At gs' age inappropriate remarks can still be funny, but the child doesn't need to know we thinks so.