Is the roommate also your landlord? You pay rent directly to him and he is not a go-between with you and the true landlord? Or do you have a rental agreement with an actual landlord?
Because, if you are both tenants with a landlord, the roommate has ZERO say in the matter. You both have equal rights as tenants.
If you are essentially the roommate's tenant, then the roommate has a little more authority, but not much. The roommate/landlord cannot legally deny you having guests. This violated your right to 'quiet enjoyment' of YOUR home.
He may RESTRICT the number of nights you have overnight guests, bur he cannot legally deny you from having people over.
So, give him three choices:
1. Tell him to let you out of the agreement without financial penalty to you.
2. Tell him to try to evict you for exercising your legal right to have guests come to visit.
3. Shut up and leave you alone.