I agree with you, but I have seen 2 sides of the story. On one hand, I usually feel sorry for them, but I've seen some who are plain out rude and deserve to be on the streets.
One time some ugly skinny dude walking a dog asked me for change, I told him I only have my card on me (which was true), he gives me a dead serious look and tells me "Want me to be a smart-@ss? I have a card reader." He said it with such a rude tone, so I told him to get the f*ck out of my face.
Then I saw one homeless guy, who begged for change and got mad because someone gave him an apple and a banana. He was mad, because he wanted change to get booze, not food.
So yeah, generally I feel bad for people having a rough time, but there are some people out there who are serious pieces of sh!t and deserve to be where they're at.