Author Topic: Bad analysis data / WXQC Mailing List?  (Read 3603 times)

0 Members and 1 Guest are viewing this topic.

Offline RickNY

  • Contributor
  • ***
  • Posts: 116
    • College Hills Weather - Farmingville, NY
Bad analysis data / WXQC Mailing List?
« on: October 16, 2018, 12:21:22 PM »
For the past several weeks, I will get QC emails reporting crazy dew point errors.. My site is E3379 -- https://weather.gladstonefamily.net/qchart/E3379
From what I can tell based on all of the other stations in my area is that the analysis data itself is wayyyy off, and not my data (or my neighbor's station data)
Looking at the dew point graphs show analysis values for dew point of 3000 degrees, 8000 degrees, etc.. Which leads me to my 2nd question -- is the WXQC Mailing list gone now?  When I try to use any of the WXQC Mailing List links, I end up with a custom 404 from Philip Gladstone's site..

Not sure what I should/could do in this instance..
Thanks,
Rick
 [ You are not allowed to view attachments ]

 [ You are not allowed to view attachments ]

Offline bchwdlks

  • Senior Contributor
  • ****
  • Posts: 196
Re: Bad analysis data / WXQC Mailing List?
« Reply #1 on: October 16, 2018, 02:02:22 PM »
It is not just your data - I looked through 8-10 'Neighboring Stations' to my station and they all have the same error.

Offline kokketiel

  • Member
  • *
  • Posts: 15
Re: Bad analysis data / WXQC Mailing List?
« Reply #2 on: October 16, 2018, 02:38:25 PM »
I'm getting the same issue, and neighboring stations show the same.  Looks like CWOP is messing up some calculations.

Offline DW7240

  • Senior Contributor
  • ****
  • Posts: 225
    • The Vicarage Weather Feed
Re: Bad analysis data / WXQC Mailing List?
« Reply #3 on: October 16, 2018, 06:00:05 PM »
Hi,

Mine and all neighboring stations around me have the same issue, it's been like this for sometime now, not sure why the issue has lasted so long - sent email to Phillip asking time line for a fix.

Will let you know the answer if I get one.

Nick. dw7240.com.


Offline RickNY

  • Contributor
  • ***
  • Posts: 116
    • College Hills Weather - Farmingville, NY
Re: Bad analysis data / WXQC Mailing List?
« Reply #4 on: October 16, 2018, 06:02:41 PM »
Seems like it happens for a stretch of 2-3 days, and then goes back to being OK.. I think its been happening for 2-3 months now.

Offline galfert

  • Global Moderator
  • Forecaster
  • *****
  • Posts: 6822
Re: Bad analysis data / WXQC Mailing List?
« Reply #5 on: October 16, 2018, 08:34:15 PM »
I'm having the same problem with dew point and humidity. Like you said it is only the analysis that is way off. I also checked all of my neighbors and I don't see any one of them being the source. For me this started a little over a week ago around October 5th.

This is very annoying. To say that it will go away seems like the wrong approach. Why hasn't gladstonefamily.net been updated in years? If the project is too much to handle or Phillip has lost interest maybe he should consider handing it over. I mean we appreciate all the work that got put into it years ago, but at some point doing the right thing for the project is just as important.
« Last Edit: October 17, 2018, 06:26:01 AM by galfert »
Ecowitt GW1000 | Meteobridge on Raspberry Pi
WU: KFLWINTE111  |  PWSweather: KFLWINTE111
CWOP: FW3708  |  AWEKAS: 14814
Windy: pws-f075acbe
Weather Underground Issue Tracking
Tele-Pole

Offline dj1111s

  • Weather obsessed
  • Forecaster
  • *****
  • Posts: 410
  • The "d" cloud.
Re: Bad analysis data / WXQC Mailing List?
« Reply #6 on: October 16, 2018, 09:42:35 PM »
Same thing going on here for quite a while now.  :-(
Vantage Pro2 Plus W/FARS, wireless anemometer, bucket heater with temperature sensor. Envoy, VP2 and Vue consoles. Meteobridge. Weatherlink Live. AirLink. Soil and leaf station. Attic temp and humidity.
Trained Storm Spotter
Member CoCoRaHS
You will never find me on Facebook or Twitter or any other social media!!!

Offline RickNY

  • Contributor
  • ***
  • Posts: 116
    • College Hills Weather - Farmingville, NY
Re: Bad analysis data / WXQC Mailing List?
« Reply #7 on: October 16, 2018, 10:37:15 PM »
I did notice that on the days this occurs, there usually ends up being big gaps in the “Analysis” data as well. For example, looking at my graphs now, I see this:

 [ You are not allowed to view attachments ]


Offline galfert

  • Global Moderator
  • Forecaster
  • *****
  • Posts: 6822
Re: Bad analysis data / WXQC Mailing List?
« Reply #8 on: October 16, 2018, 10:45:07 PM »
It is probably due to the 2001 Pentium 4 CPUs. Surprised they lasted this long.
Ecowitt GW1000 | Meteobridge on Raspberry Pi
WU: KFLWINTE111  |  PWSweather: KFLWINTE111
CWOP: FW3708  |  AWEKAS: 14814
Windy: pws-f075acbe
Weather Underground Issue Tracking
Tele-Pole

Offline kokketiel

  • Member
  • *
  • Posts: 15
Re: Bad analysis data / WXQC Mailing List?
« Reply #9 on: October 17, 2018, 12:37:14 PM »
I did notice that on the days this occurs, there usually ends up being big gaps in the “Analysis” data as well.

I don't think the gap is in the data - it's the dewpoint that is way off the scale...

Offline RickNY

  • Contributor
  • ***
  • Posts: 116
    • College Hills Weather - Farmingville, NY
Re: Bad analysis data / WXQC Mailing List?
« Reply #10 on: October 17, 2018, 12:43:28 PM »
If you notice on my first pic, the graph does adjust scale when the analysis data is off.

Offline galfert

  • Global Moderator
  • Forecaster
  • *****
  • Posts: 6822
Re: Bad analysis data / WXQC Mailing List?
« Reply #11 on: October 17, 2018, 03:52:06 PM »
It isn't just the graphs with thousands of degrees analysis that is off it is also the analysis data.

 [ You are not allowed to view attachments ]

This message brought to you by Gladstonefamily.  Red Xs for everyone !

 [ You are not allowed to view attachments ]
Ecowitt GW1000 | Meteobridge on Raspberry Pi
WU: KFLWINTE111  |  PWSweather: KFLWINTE111
CWOP: FW3708  |  AWEKAS: 14814
Windy: pws-f075acbe
Weather Underground Issue Tracking
Tele-Pole

Offline SlowModem

  • Weather at the speed of dialup!
  • Forecaster
  • *****
  • Posts: 6641
  • WX @ 26.4 kbs
    • Watts Bar Weather
Re: Bad analysis data / WXQC Mailing List?
« Reply #12 on: October 18, 2018, 03:34:03 AM »
It is probably due to the 2001 Pentium 4 CPUs. Surprised they lasted this long.

LOL I've still got a Dell Deminsion 2300 uploading my data flawlessly since 2007 (but I purchased it in 2002) Serial connection data logger and all.
Greg Whitehead
Ten Mile, TN USA

Offline DW7240

  • Senior Contributor
  • ****
  • Posts: 225
    • The Vicarage Weather Feed
Re: Bad analysis data / WXQC Mailing List?
« Reply #13 on: November 03, 2018, 11:28:59 PM »
Just for the record, Dew point is back to normal, thanks Phillip for your efforts - Be interested to know what the issue was, but it's fixed - that's what counts.

Nick. dw7240.com


Offline galfert

  • Global Moderator
  • Forecaster
  • *****
  • Posts: 6822
Re: Bad analysis data / WXQC Mailing List?
« Reply #14 on: November 04, 2018, 08:07:03 AM »
Thanks Phillip? For what? I don't know that Philip has done anything recently. He certainly hasn't communicated anywhere that I know of. For all we know the problem fixed itself only because its been a few weeks without issue and maybe the servers are not glitching. Or maybe the hackers got tired of messing with old Pentium 4 hardware. There is no comfort in not knowing what happened, what was done to fix it if anything, and the uncertainty that the lack of communication would mean that any day it could happen again if the system is running without any maintenance. If you look at historical data the errors are all still there. So it hasn't been cleaned up.

I've used online tools to see what software version these servers are running and they haven't been patched nor upgraded in many years. This is no way to run servers that are exposed to the Internet that contain known security holes and unpatched exploits. The website contains numerous dead links. The website has not changed in forever. I've sent emails that get no response. And these have been emails not complaining about problems but with simple questions and with closing remarks that I would be sending in PayPal contribution money. No response, no thank you for your support. Philip is MIA.

I can't thank Philip enough for what he built but if maintenance is not possible after all this time then it is time to pass the baton like Steve of Cumulus has recently done.
« Last Edit: November 04, 2018, 08:22:56 AM by galfert »
Ecowitt GW1000 | Meteobridge on Raspberry Pi
WU: KFLWINTE111  |  PWSweather: KFLWINTE111
CWOP: FW3708  |  AWEKAS: 14814
Windy: pws-f075acbe
Weather Underground Issue Tracking
Tele-Pole

Offline ocala

  • Forecaster
  • *****
  • Posts: 4383
Re: Bad analysis data / WXQC Mailing List?
« Reply #15 on: November 04, 2018, 09:37:33 AM »
I don't think you should pass judgement. We don't know the why's/ what's/ or where about's of Phillip. Until that information becomes public just assume that it's business as usual. Besides, it's a free site. If you don't care for the service then don't send your data.
The blues had a baby and they named it Rock & Roll

Offline Otis

  • (aka Paul)
  • Forecaster
  • *****
  • Posts: 724
    • Lake Huron Weather
Re: Bad analysis data / WXQC Mailing List?
« Reply #16 on: November 04, 2018, 11:42:50 AM »
I don't think you should pass judgement. We don't know the why's/ what's/ or where about's of Phillip. Until that information becomes public just assume that it's business as usual. Besides, it's a free site. If you don't care for the service then don't send your data.

+1

CW3699

Offline galfert

  • Global Moderator
  • Forecaster
  • *****
  • Posts: 6822
Re: Bad analysis data / WXQC Mailing List?
« Reply #17 on: November 04, 2018, 08:55:43 PM »
I don't think I'm passing judgment. I only asked why anyone would thank Phillip for doing anything recently when there is no evidence that anything was done. Then I listed a bunch of facts. Yes I also threw in some speculation about hackers and Pentium 4 servers, but that is not passing judgment. I'm just frustrated with how the service is run. I'm voicing my displeasure so that I can raise awareness and find a following and create buzz and offer solutions to hopefully create improvements. I'm a supporter of the service so I have no interest in abandoning what I want to see improved. If I buy a car and after owning it for a while I find out something is not to my liking that wasn't immediately obvious when I bought it I'm going to want to offer constructive criticism and feedback to cause improvement and then when I trade in I might have better choices. The service may be free to some, but some people are donors, and donors want to be recognized, have their feedback heard, and thanked for their support. That isn't happening.

If anything I said can be twisted and be called passing judgement, then a "Thanks for fixing" is a judgement that something was done, without having any information to base that response. Goes both ways. I just wanted to point out that nothing has been maintained in years with proof...facts. Therefore I feel justified to pass judgment on those facts. So depending on how you look at it I may or may not be passing judgment and the same goes for the baseless thank you.

If you want to assume it is business as usual I agree with with you. It does seem like business as usual, which is inaction, and no response to the community.

This is getting to be a bit more confrontational than I would like. I hope that we can move along and find ourselves on the same side. I love being a part of CWOP and I would like to see it improved.

There are some people that are haters and like to bash other things that they don't like nor support. They could be called trolls. Jump on those people by all means (or ignore them some people say, don't feed the trolls). But I'm not one of those. If I came off that way then I'm sorry.
« Last Edit: November 04, 2018, 09:06:12 PM by galfert »
Ecowitt GW1000 | Meteobridge on Raspberry Pi
WU: KFLWINTE111  |  PWSweather: KFLWINTE111
CWOP: FW3708  |  AWEKAS: 14814
Windy: pws-f075acbe
Weather Underground Issue Tracking
Tele-Pole

Offline jmcmurry

  • Jim McMurry
  • Forecaster
  • *****
  • Posts: 528
  • Davis Vantage Pro 2 Plus Wireless.
    • Juneau County Weather
Re: Bad analysis data / WXQC Mailing List?
« Reply #18 on: November 05, 2018, 05:29:46 AM »
I've always found Philip to be very responsive when I've contacted him.  I worry that something may have happened to him.

- Jim

Forum Search and Google Can be Your Best Friends

Offline galfert

  • Global Moderator
  • Forecaster
  • *****
  • Posts: 6822
Re: Bad analysis data / WXQC Mailing List?
« Reply #19 on: November 05, 2018, 06:04:15 AM »
I do see evidence that Phillip was very involved at one point and he obviously cared.. I wish that someone that knows him could reach out and see how he is doing. I think that there are lots of people that would be willing to help.

Ecowitt GW1000 | Meteobridge on Raspberry Pi
WU: KFLWINTE111  |  PWSweather: KFLWINTE111
CWOP: FW3708  |  AWEKAS: 14814
Windy: pws-f075acbe
Weather Underground Issue Tracking
Tele-Pole

Offline W Thomas

  • Welcome to my area!
  • Forecaster
  • *****
  • Posts: 1643
  • In Loving Memory Of Hooker The Weather Dog !
    • Smyth-Grayson Weather
Re: Bad analysis data / WXQC Mailing List?
« Reply #20 on: November 27, 2018, 08:13:10 PM »
It is probably due to the 2001 Pentium 4 CPUs. Surprised they lasted this long.

LOL I've still got a Dell Deminsion 2300 uploading my data flawlessly since 2007 (but I purchased it in 2002) Serial connection data logger and all.

Dimension 4700 here on XP even!  A little slow but real reliable!


     Best Regards
     Wayne

CWOP CW8217
KVAWHITE22 Wunderground   Davis VUE &  Davis Vantage Pro 2  /   Dedicated Server
GR Level 3 ,Level 2 AE Radars  Weather Display 10.37P  Mid Atlantic Weather Network Member
SkyWarn & Spotter Network 6092

Offline RickNY

  • Contributor
  • ***
  • Posts: 116
    • College Hills Weather - Farmingville, NY
Re: Bad analysis data / WXQC Mailing List?
« Reply #21 on: December 14, 2018, 11:49:13 AM »

I see this issue seems to have returned.. Past couple of days, my error reports are showing dew point errors of 80-150 degrees.

Offline RickNY

  • Contributor
  • ***
  • Posts: 116
    • College Hills Weather - Farmingville, NY
Re: Bad analysis data / WXQC Mailing List?
« Reply #22 on: December 14, 2018, 01:49:24 PM »
1) MADIS is computer program that adheres to the adage "Garbage IN, Garbage OUT."
2) It only takes ONE 'bad apple' wx station to "spoil" the MADIS sample pool.
3) Most wx people do NOT check/compare what they are sending against what MADIS expects to find.
4) In Real Estate it's Location--location--location, in home wx stations it's Microclimate--microclimate--microclimate.
5) These are HOME wx stations NOT NOAA/FAA certified stations, that's why we can afford them.

And normally that would be the case -- however, when this has happened over the past few months, very widespread errors are shown in the data across the entire country.  For example, I'm in NY.. If I check stations in Missouri -- I see similar dew point errors reported.  Its not from a single station thats causing the sample data to be wrong, or even multiple wrong stations in a close area.


Offline RickNY

  • Contributor
  • ***
  • Posts: 116
    • College Hills Weather - Farmingville, NY
Re: Bad analysis data / WXQC Mailing List?
« Reply #23 on: December 14, 2018, 02:51:40 PM »
The NOAA/NWS Meteorological Assimilation Data Ingest System (MADIS) is NOT just for local wx, it is *THE* data collecting (and analysis) system that feeds the WHOLE USA weather mapping program. We just "see" it's local results based upon whatever stations it has selected as its "pool", which changes as the inputs change; it's NOT always the same stations being sampled and used.

Some common "input" problems that MADIS contends with are: different timestamps, units, wrong elevations, incorrect locations (lat & long), etc.

I think it would be reasonable to expect that the whole system would reject as invalid dewpoints of 20,000 degrees.. And not flag a slew of data as not meeting QC because it was 19,950 degrees low.. Just saying..

Offline galfert

  • Global Moderator
  • Forecaster
  • *****
  • Posts: 6822
Re: Bad analysis data / WXQC Mailing List?
« Reply #24 on: December 14, 2018, 04:21:45 PM »
Old Tele man,
This is not a MADIS issue that RickNY is reporting and as others have seen. You can look at the MADIS analysis and the results are fine. The issue being reported only has to do with Gladstonefamily.net. That is a poorly run system that ingests MADIS data and then does its own separate analysis. That is where the problem is ....not MADIS.

But to your credit you are correct that there are times when MADIS does ingest bad data from bad stations. But never have I seen MADIS report thousands of degrees of error.  MADIS is very good at throwing out really far off bad data. MADIS does make minor errors as you are reporting though. Unfortunately Gladstonefamily hick-ups or something with its own far off data analysis nationwide for no apparent reason sometimes.
Ecowitt GW1000 | Meteobridge on Raspberry Pi
WU: KFLWINTE111  |  PWSweather: KFLWINTE111
CWOP: FW3708  |  AWEKAS: 14814
Windy: pws-f075acbe
Weather Underground Issue Tracking
Tele-Pole