I tend to agree with nincehelser on this. In my experience, wind data is inherently
very noisy, even in nearly calm conditions. Your noisy data is not necessarily an indication of a problem with the wind vane. That's not to say there isn't a problem...but the noisy data does not by itself prove the existence of a problem.
You might check to see if the vane is balanced. Try holding the 5n1 sensor horizontal (90 degrees from the normal mounting orientation). If the vane tends to spin around to the same orientation after you move it, then it's not balanced. The thing is that I am not 100% certain that it
should be balanced. But if it is not balanced and that's intentional, then the wind vane shaft must be mounted perfectly vertical -- tilt it and you will have problems.
Perhaps there are some wind vane experts here that know if it should be perfectly balanced? My 5n1 is NOT perfectly balanced and will spin around to "vane down" if held horizontally.
The only other thing I can offer is an observation that wind vane design is far from trivial. If you want to get a sense of what's involved, have a gander at this paper which looks at some of the technical aspects involved.
http://journals.ametsoc.org/doi/pdf/10.1175/1520-0450%281967%29006%3C1114%3AEADOWV%3E2.0.CO%3B2