Same reason that Valentines Day can be hard. People have a picture in their minds of an ideal version of those days, and when the reality doesn't compare well, they feel it as a loss. In fact, you might have a perfectly good ordinary day, but on certain calendar dates, that feels like it isn't good enough. We want more.
This is especially true as it pertains to family relationships. We wish for our holidays to be spent with family members who accept and love us as we are, and enjoy the same traditions that we enjoy. In reality, our family may be dysfunctional, or unavailable due to distance or death, or may not have ever existed at all, in the form we have in our mental picture.
Holidays can still be a time of joy, if we learn to appreciate what we can have, and let go of any detailed expectations.