Well, etheltbags, I fully understand how you feel, but for me dressing well is about self-respect.
What you wear always says something about you as a person. It has to, because the person putting the clothes on has decided to be a careful, dresser, or, like you, someone who just puts the nearest garment on, but even then the wearer will have chosen to buy/accept that garment into their wardrobe so that it is available for wear and other people seeing them will reach conclusions about who and what they are from their appearance. You cannot avoid it.
As far as I am concerned I love dressing and looking nice. It is a sure sign that I am unwell or very tired if I am wandering round the house in a tracksuit looking unkempt. It is probably no co-incidence that I also like my house to look nice. I choose my soft furnishings, paint/wallpaper and everything in each room with care. In clothes I think make me look pleasant and sitting in a living room where everything around me is chosen with care in colours and patterns I like and complement each other I feel relaxed and happy, in a dingy old room with dingy clothes, I become equally dingy and withdrawn.
I have had a delightful morning. Since last winter I have lost nearly 2 stone. This morning I got my winter clothes out and tried them all on. They are all far, far too big for me and will need replacing. Yippee, I am so going to enjoy going out to buy a whole lot of new clothes.
I see no reason whatsoever for having to be apologetic for enjoying having a wardrobe of clothes, chosen with care, which I delight to wear. Its my money I can spend it on what I like - and I like clothes.