It's not fear or guilt, at least not in this case.
It is simply the human bent towards negativity.
Let me illustrate:
You are at a function and think you look pretty good, in fact several people have admired your outfit and said so. You're smiling.
Then an acquaintace sidles up to you and whispers, "What an earth made you wear that? It doesn't suit you at all and the colour is dreadful.'
Do you go by the 5 people who said you look great and continue in buoyant mood, or immediately want to go home and change?
Of course, there are those who will just shrug it off, at least on the outside and pretend they don't care but most people will immediately feel less than their best and wrong-footed.
We are somehow programmed to remember our own negative traits and perhaps it's to remind us to curb them if we can and dwell more on the positive.
Of course if we are egged on to continue complaining by others (ahem, ahem) we get worse.