WXforum.net

Weather Related Organizations => WeatherUnderground => Topic started by: frankpc on March 13, 2019, 04:33:53 PM

Title: Station requirements to satisfy WU's QC
Post by: frankpc on March 13, 2019, 04:33:53 PM
I have been using sensors at my home for solar radiation, outside temperature, and soil moisture for three years now.  I use the data to help calculate evapotranspiration each morning for lawn irrigation.  I'm in a position to publish the data to WU but I don't know what other sensors would be required to meet their requirements. 

I would rather not buy a complete weather station since I already have several of the sensors functioning and can obtain additional sensors as required.

But it isn't clear to me what sensors are required to meet WU's QC standards.  Is that documentation posted on this forum?

I do not have sensors for wind direction, wind speed, pressure, humidity, precipitation, and dew point.

Thank you 
Title: Re: Station requirements to satisfy WU's QC
Post by: Cutty Sark Sailor on March 13, 2019, 08:10:18 PM
WU...QC ?...rotflmya..

Title: Re: Station requirements to satisfy WU's QC
Post by: WSWeather on March 13, 2019, 08:16:48 PM
Nice idea, but you can't.

Title: Re: Station requirements to satisfy WU's QC
Post by: frankpc on March 13, 2019, 11:28:07 PM
Nice idea, but you can't.

You are saying that you can't construct your own PWS and have it accepted by WU? 

One of the device choices in the list of WU's hardware is "Raspberry Pi".  I took that to mean that WU approved of building your own PWS as long as the results met their QC standards.  (don't laugh Cutty) :-) 

actually, I'm confident that a DIY PWS could easily be more accurate than many of the other less expensive weather stations.  What am I missing?

Thanks for the help.
Title: Re: Station requirements to satisfy WU's QC
Post by: CW2274 on March 13, 2019, 11:38:51 PM
Nice idea, but you can't.
as long as the results met their QC standards. 
They have none. If you can find a way to send them what you've got and fake the rest, they'll post it. 
Title: Re: Station requirements to satisfy WU's QC
Post by: Cutty Sark Sailor on March 13, 2019, 11:50:49 PM
Nice idea, but you can't.
as long as the results met their QC standards. 
They have none. If you can find a way to send them what you've got and fake the rest, they'll post it. 
WU...QC ?...rotflmya..
Yep...
Title: Re: Station requirements to satisfy WU's QC
Post by: WSWeather on March 14, 2019, 12:01:35 AM
Nice idea, but you can't.

You are saying that you can't construct your own PWS and have it accepted by WU?

You can do whatever you want, but there is no "QC" target to hit, nor is there any assurance they will accept anything you toss their way.  WU has accepted complete trash and ignored spot-on highly accurate data.

Good luck.
Title: Re: Station requirements to satisfy WU's QC
Post by: Cutty Sark Sailor on March 14, 2019, 12:04:11 AM
Nice idea, but you can't.

You are saying that you can't construct your own PWS and have it accepted by WU?

You can do whatever you want, but there is no "QC" target to hit, nor is there any assurance they will accept anything you toss their way.  WU has accepted complete trash and ignored spot-on highly accurate data.

Good luck.

Nice idea, but you can't.
as long as the results met their QC standards. 
They have none. If you can find a way to send them what you've got and fake the rest, they'll post it. 
WU...QC ?...rotflmya..
Yep...

Catching on yet?
Title: Re: Station requirements to satisfy WU's QC
Post by: frankpc on March 14, 2019, 12:50:38 AM
Quote
If you can find a way to send them what you've got and fake the rest, they'll post it.

What I want to do is to build the weather station which would consist of whatever sensors they require.  I was reading this forum earlier about PWS owners who have had their stations disconnected for what seems to be a non discernible reason.  So I figure there must be a set of measurements that are mandatory.  I know a lot of stations do not have solar radiation and soil moisture.  Some stations are advertised to be 5 in 1 and others are 10 in 1.  And from the list of hardware, both are acceptable.

I was hoping to get this figured out first before I invest time into this thing. 

Thank you for the advice.  I appreciate it!
Title: Re: Station requirements to satisfy WU's QC
Post by: frankpc on March 14, 2019, 12:52:27 AM
Quote
Catching on yet?

Yeah.  No one ever said I was quick.

:-)
Title: Re: Station requirements to satisfy WU's QC
Post by: frankpc on March 14, 2019, 08:13:19 AM
Perhaps my ultimate goal is different than most.  I need data for evapotranspiration and to water the lawn.  In order to do that, I need accurate precipitation information in my immediate area.  The only service that has successfully done that for the past few years is WU.  The advantage WU has is that there are several WU PWS's in proximity to my house, so the precipitation data is applicable. 

Very likely, all of the other data needed for the evapotranspiration calculations are obtainable from other services.  But precip is unique to my area.

Is there another service online that provides precip info from personal wx stations?  There very well could be!  That would be great!  I haven't had a need to search for a source other than WU before.

I appreciate your time and advice.  Thanks!
Title: Re: Station requirements to satisfy WU's QC
Post by: PaulMy on March 14, 2019, 11:16:26 AM
Quote
Is there another service online that provides precip info from personal wx stations?  There very well could be!  That would be great!  I haven't had a need to search for a source other than WU before.
There are plenty - PWS, AWEKAS, Weathercloud, WOW, CWOP, but none have the same numbers of stations and the same access to the posted data.  And of course CoCoRaHS has the daily manual precipitation readings.


Enjoy,
Paul
Title: Re: Station requirements to satisfy WU's QC
Post by: havtrail on March 14, 2019, 12:14:36 PM
I have been using sensors at my home for solar radiation, outside temperature, and soil moisture for three years now.  I use the data to help calculate evapotranspiration each morning for lawn irrigation.  I'm in a position to publish the data to WU but I don't know what other sensors would be required to meet their requirements. 

I would rather not buy a complete weather station since I already have several of the sensors functioning and can obtain additional sensors as required.

But it isn't clear to me what sensors are required to meet WU's QC standards.  Is that documentation posted on this forum?

I do not have sensors for wind direction, wind speed, pressure, humidity, precipitation, and dew point.

Thank you

You'll have to forgive some people's pent-up negativity towards the present state of WU, which seems to be getting in the way of direct answers to your questions. You came on stage here several years into a real soap opera...    :sad:

I have some info that may help. A PWS (https://www.wunderground.com/personal-weather-station/dashboard?ID=KPAARDMO5 (https://www.wunderground.com/personal-weather-station/dashboard?ID=KPAARDMO5)) near me posts to WU regularly. It lacks any barometer, precipitation, wind and solar data, yet has been on WU as long as I have been looking. Don't know if you need humidity data or not, since most stations use a combined temp/humidity sensor. I believe dew point can be derived from temperature and humidity by WU, which also adds UV to my station's data even though I do not report it.

I would say go ahead and try to publish to WU with what you have, and see how it goes. One thing that's been made clear in the responses is you don't need to worry yourself much about any QC problems with WU. You may face some delay and have a quirky issue or two with getting registered and data showing up, and that may not be because of any QC issue or the lack of some data parameter. Try it, persist with it, and see what happens.

Rich K.
Title: Re: Station requirements to satisfy WU's QC
Post by: frankpc on March 14, 2019, 01:15:03 PM
Try it, persist with it, and see what happens.

Thanks Rich. I appreciate the support.  I'll get a script written up to send the data for the sensors I am monitoring now.  It'll be worth a shot.   I'll add additional sensors as I get them.  I hope to have a tipping bucket connected up in a few days.  Windspeed and direction may have to wait.

Frank
Title: Re: Station requirements to satisfy WU's QC
Post by: quailvalleywx on March 14, 2019, 02:28:23 PM
Is that documentation posted on this forum?

This is the only document I know of but it doesn't list minimum information required nor error tolerances.  Repeating what havtrail said, you will likely just have to try it.
https://feedback.weather.com/customer/en/portal/articles/2924682-pws-upload-protocol?b_id=17298 (https://feedback.weather.com/customer/en/portal/articles/2924682-pws-upload-protocol?b_id=17298)

Is there another service online that provides precip info from personal wx stations?

This will get you surrounding CWOP station data http://www.findu.com/cgi-bin/wxnear.cgi?call=KA2FNK (http://www.findu.com/cgi-bin/wxnear.cgi?call=KA2FNK).  If you are in Shawnee, KS my station is South of you - FW3280.  I post the same CWOP data plus soil moisture to WU - see post signature for links.  Oh, WU does not archive soil data so maybe that further taints utilizing them.
 
Title: Re: Station requirements to satisfy WU's QC
Post by: frankpc on March 14, 2019, 03:40:36 PM
Great news!  I'm near Woodland and Johnson Drive.  I wasn't aware of findu.com or CWOP.  Do they allow downloading of formatted data like WU does?

I'm going to work on getting my sensors uploaded to WU then I'll take another look at those sites.

Thanks!
Title: Re: Station requirements to satisfy WU's QC
Post by: CW2274 on March 14, 2019, 03:57:12 PM
Basically the same data wise, however CWOP is much more exclusive compared to WU, if just simply for the amount of PWS's. CWOP is certainly more reliable, and as a whole, I trust the data more so (MADIS and MesoWest actually have working QC).
Title: Re: Station requirements to satisfy WU's QC
Post by: galfert on March 14, 2019, 04:06:42 PM
No sense in reinventing the wheel. Use WeeWx and add all the sensors you currently have. Then publish your data to local SQL. You don't need to publish your data online to then bring it down if you already have it locally. But you can certainly use WeeWx to publish to just about any online service also.

After taking a look at basic WeeWx setup, take a look at this regarding adding your sensors.
https://github.com/weewx/weewx/wiki/add-sensor
Title: Re: Station requirements to satisfy WU's QC
Post by: frankpc on March 14, 2019, 10:24:02 PM
Thank you for the help! 

I now have a PHP script that uploads my solar radiation and soil moisture to my WU wx site.  And the data displays.  It's a miracle.  More sensors on the way.

Is the api key to download the data sheets from WU different than the key used for the wx site?  If so, how do you get the second key?

Thanks again,

Frank
Title: Re: Station requirements to satisfy WU's QC
Post by: galfert on March 14, 2019, 10:58:40 PM
Yes there is a separate API to get your data from WU and requires process to get that API key also.

See this thread for more info.
https://www.wxforum.net/index.php?topic=36260.0

Title: Re: Station requirements to satisfy WU's QC
Post by: frankpc on March 15, 2019, 12:26:02 AM
Got the new API key.  Thanks!

Seems we have shifted to JSON for our downloads of forecast weather.
Title: Re: Station requirements to satisfy WU's QC
Post by: wunderground-PWS on March 25, 2019, 09:59:33 PM
Hi Frank, didn't read through the whole thread, too much hate!

Here's how we perform QC

steps
1) Sanity, can this temp, dewpoint, precip amount... possibly exist on earth
2) temporal check, is the station doing something funny that makes it unlikely the readings are correct (mainly, is the value not changing at all)
3) spatial check. this one only applies to temperature,  how well does your temperature compare to that of your neighbors


What dont we do with this info?
We don't do  too much with this for the web site.  The PWS dashboard pages are multi use, they are there for public consumption and also for the owner of the PWS to see the data they provide.  We should do more to surface the QC checks we perform

What DO we do
1) goldstar if you pass all quality control checks for EVERY Temperature observation you send for 5 days (or on some cases less) you get a gold star. If you send 1 bad ob, the clock is reset.

2) maps  If you are sending bad data, your station will not display on the wundermap, or even the PWS dashboard page map. 



Title: Re: Station requirements to satisfy WU's QC
Post by: frankpc on March 25, 2019, 10:18:29 PM
Thank you.  Great information.  I just now installed a new temperature sensor in a new enclosure and I notice it is more in line with neighboring sites than my previous sensor.  It had been off 1.5 or so consistently.

Glad the services are coming back up now.

Thank you for your hard work on this.

Frank