Yes. When I have people help me, I am always telling them to watch what they say cause the neighbors can usually hear you. Nothing worse than one of your employees talking about (you know what) and then looking over to see someones daughter playing in the front yard!
Anyways, it does leave a bad impression on your business, I think.
I agree that it may be a detriment, but whenever I've been working with our guys on a bigger landscaping job, we tend to joke around and listen to music etc. It lightens up the mood and makes work more fun. One of the reasons I love my job so much is the people I work with and the atmosphere of it, something I could never get sitting at a desk.