You could have kept a small, well trimmed beard, and that would be fine.
Otherwise, it was probably right. People do get paranoid around an
obviously Muslim looking guy who wears a long bushy beard.
People who think that's 'racist' and tell you not to worry about it
don't have to live in your shoes. They toss off their comments
without considering the consequences. Once, in my neighborhood,
there was a Muslim guy who didn't speak very good English, and
he was loitering around a daycare center waiting to pick someone
up. Boy, those ladies there called the police on him in a heartbeat.