There are a lot of things that everyone pays for, but not everyone uses. Some people don't read books; or have no interest in the arts; or are not into tennis, football, keep fit, swimming, etc. But everyone contributes towards the cost of libraries, council sports facilities, the opera/ballet, museums, art galleries, etc. etc. Some people believe that there should be no subsidy of the arts - and that is a valid point of view (although I absolutely disagree with it) - but I don't understand why one particular form of entertainment/information/education should be singled out as an unfair "tax".
I do, however, agree that it seems a bit much to criminalise people for not having a TV licence. I suppose the reasoning behind this is that if there was not a strong deterrent, a lot more people would risk not paying.