So you were walking 5 hours and didn't even have a coffee. Not saying he should buy both but that sounds like a cheapskate to me anyway. And then he asked you for cash! I do think if you hadn't lent it to him you might not have seen him again, or he might have turned nasty. All you actually know about him is what he has told you - it could all be rubbish. And the £20 sort of means he has a ready-made reason to meet you again even if you didn't want to. I think go with your instincts and don't meet him again, say he can keep the £20 and block his number.
OH lent a fairly newish friend money which we didn't see again. Not a fortune, but we had a young family and it was a lot for us. Turned out, when he disappeared, that quite a few people had lent money to him. I didn't like him much anyway so decided the money was the price we paid for getting him out of our lives and put it out of my mind. I think £20 is a reasonable price to pay for getting this man out of your life.