I'd say "yes." Because some anime voices can be REALLY annoying in either circumstance. Naruto for instance; I first watched it dubbs only; when I heard the English voice, I was just like.. No. I thought the Japanese voice fit perfectly for Naruto's character and that the English just seemed miss-matched.
I couldn't vote because what I wanted to say wasn't there.
It depends. I've seen the entire Inuyasha series in English, but when I watch them on DVD, I watch with subtitles, because really, the translations often leave something to be desired. Not only that, but often, the Japanese VA's are MUCH better. On the other hand, I kinda enjoy some of the English VA's, like Bleach's (well, most of them).
So sometimes I will watch in English, most of the time, not. I give everything a chance, because absolution is retarded.