Many police departments wouldn't respond to a call about someone smoking weed, in and of itself. Reports that you were dealing or storing large volumes of it might get their attention, but many departments aren't interested in simple use in and of itself.
To answer your question about invasion of privacy, though, it isn't one. If your neighbors can see you from their own yard, where they have every legal right to be, then legally that's fair game. Basically, if your activities are visible in "plain sight" from a public area, or somewhere where the other person is legally allowed to be, then that's not an invasion of privacy.
If your neighbors had used an infrared camera to see through your walls or something, then you might have a point, but if they can see you unaided, generally not.