I think that what you say is true, Lathyrus, and try to 'check my privilege' when I comment on things like this, as I am aware that I don't have an asylum hostel on my street, I am not waiting to be housed and my children aren't having their education held back because English is a minority language in their school.
I feel the same when people comment about other things - how more children should have apprenticeships (just not theirs, who should of course go to university), how giving WFA to people on pension credit is making sure that it goes to those who 'need' it, when they aren't paying rent and struggling on a tiny pension that takes their income a few pounds over the mans-test limit, or how dreadful it is that people look for ways to ensure they can leave something to their children if they need care, when the complainers have enough to pay for care and still leave a lot behind. Many policies that rely on means-tests are deeply unfair, as is those who 'have' deciding on what others 'need'. I complain about that a lot on here.
People do have a right to feel aggrieved about all of those things (and more), but the answer is, IMO, to deal with the reasons why the inequalities exist, rather than blaming the people who need assistance.
Raise minimum wage so that people in full-time work don't have to rely on benefit top ups. Make pension contributions compulsory, and if people don't pay in they shouldn't get one unless they have been unable to (as opposed to not wanting to) work. Give everyone an education fund that they can draw on at any time of their life, so that if someone wants to leave school early, work for years then retire and study poetry or floristry that is fine, as is getting a degree at 22 and a professional qualification at 45. Have a ring-fenced tax for social care, so that anyone who has paid it gets care without charge. That sort of thing.
What we do about people who slip through the net is for us (As a country) to decide, but the expectation should be that if you don't contribute you don't benefit from the fruits of others' labours. If people have to think carefully about what sort of society they want to be part of, they might feel less punitive when it comes to other groups. We could go the way of the US (and much of the developing world) where if you don't work you starve, and if you are ill the treatment stops when your money runs out. Or we could opt for a more humane system that expects all citizens to contribute when they can so that they are 'covered' when they cannot, such as we supposedly have now. The policing of that system needs to be tweaked, to ensure that it is properly reciprocal, and a good place to start would be to allow asylum seekers to work and pay tax, which would then allow them to pay rent and so on, so they don't need to be given priority over others. People may then be less resentful of them.
Meanwhile, however, there is nothing unreasonable about being aware that anti-immigrant propaganda is pernicious, that many of the figures we are given are false, that there are reasons why asylum seekers hang about their home towns (they are not allowed to work and have no money for recreational pursuits) and that if they are vulnerable they have housing needs in the same way that vulnerable British people do.