Feelings aren't wrong. You may feel however you want to feel about the situation.
It's OK to tell him that you aren't happy in your current town and would like to talk about moving.
It's OK to express frustration that you guys never see each other because of work, and ask him to help you think up a new plan where you can see each other more and both make enough money to get by.
It's OK to break up with him. You don't have to stay together just because you've been together a long time. You don't have to stay with him just because he's a good guy. You can love someone very much but still wind up breaking up because it's just not working out.
If both partners agree to it, there's nothing wrong with a relationship where they only see each other every so often, and otherwise lead separate lives. But, again, you BOTH have to agree to that, and it doesn't sound like he wants that.
Think about some ways you can possibly fix this problem, talk to him about it, see if he's interested in a change or compromise, and go from there. If you guys are unable to find an answer or if he's not interested in changing things up, then you may have a difficult decision ahead of you.