If 'rich' people or those whose lifestyle and income means they never apply for a pass or never use it, there is no loss to the economy.
I met two ladies, friends, who looked a bit more senior to me, in the M&S food hall, very nicely made up and well dressed unlike me sans make-up and wearing Primark jeans and sweatshirt.
I initiated a conversation whereby they told me they met every weekday to lunch in M&S or JL and often to dit a bit of shopping. They travelled by car then caught the Guildford Park and Ride bus.
I had not driven but went the whole journey by bus. The bus stops right in town anyway. I would be happy to pay for a day rider ticket or a yearly bus pass but, as others have said, removing it indiscriminately might disadvantage those whose need it most.