I have always peeled them like my mother did.
I've never worked out how wiping them with a dry cloth or kitchen paper can do anything but smear any dirt just further around the mushroom. Washing certainly spoils the taste, although I do worry as to what might be hiding between those gills. One German chef recommended sprinkling flour into a bowl of water and quickly washing them in that - the theory being that the flour picks up the dirt. I tried that but it didn't work for me, so it was back to peeling.
I only shop once a week, so sometimes I've had them a few days and they have become a bit unsightly which is another reason I prefer to peel them.