Don't worry about it, if you deny them access to being in your dreams, then you'll be fine.
In whatever way works best for you, close off your personal dream space with a metaphorical "Do Not Disturb" sign outside the domain of entry.
You have your personal dreams where only you or people you allow in can visit, or you have the communal "public space". If you want to just dream without anyone visiting you, then stay in your own head for your dreams without just letting anyone else in. In can be as easy as reminding yourself "I'm not letting anyone else in my own dreams tonight" before going to sleep.