I understand your concern, but I personally believe that it represents a misunderstanding of how the economy works with human nature. This is what I mean:
The space programs were not conducted by government bureaucracies. Sure, NASA was involved, but only in an administrative and organizational role. Most of the technological effort was actually performed by private industry under contract to the government. Space suits, rocket engines, re-entry tiles - all those things were provided by people working for a living at their local aerospace factory. Did that "improve life here on Earth?" Well, it certainly did for the hundreds of thousands of folks that were involved in the effort, and the folks that owned the restaurants that they frequented, and the folks that built the houses that they lived in. Furthermore, the space programs actually succeeded. It's not like we have to repeat the Apollo series endlessly. We have been there and done that (despite whacko internet claims to the contrary).
Now let's compare that with government programs that were created with the express purpose of "improving life here on Earth." The War on Poverty was initiated in the mid 1960s. It has continued in one form or another for decades, and we continue to pour trillions into it. It has not succeeded. We have more poverty now than we did then, and it has cost thousands of times more than the NASA budget for all those years. How many more decades do we have to devote to that futility?
Your example of a hypersonic transport is comparable. NASA is working on that technology. Great. It is being accomplished by folks working in private industry. But the technology is not the issue. The free market is the issue. You cannot force business folks to endure two hours of discomfort and charge them $20,000 for a single trip across the Atlantic. They are not gonna do it. That's not "improving life on Earth." I guess you could subsidize their tickets, but that's not "improving life" for the taxpayers who have to pay the guy's fare to take care of his own business.
There is a difference between exploration and free enterprise. Exploration involves things that private industry would never undertake on their own, because there is no immediate profit motive. If that exploration takes us to the planets, the goal is sufficient unto itself, especially for the minimum cost and the economic benefits it provides in the form of private business.
Free enterprise occurs after the technology is complete. You cannot force it to occur. That's what we keep trying with massive government incentive programs. Green energy, hypersonic transports, electric cars, etc. You cannot force someone to buy an electric car for $45,000 because you think it will help clean up the planet (especially if it won't). And if you help him pay for it by raising taxes on everyone else, you are just kidding yourself. That won't make it any more attractive, and it will not drive the price down.
If you eliminate the space programs, you do not help anyone. You just hurt the folks that worked on the programs. It is not a zero-sum game. You do not get to simply move that money elsewhere. I wish it worked that way, but it does not. The money is lost along the way in bureaucracy and administration. You end up with - essentially - nothing. That would not be a wise choice.
A lifelong career working in private industry with government funded contracts