Author Topic: Bad analysis data / WXQC Mailing List?  (Read 1958 times)

0 Members and 1 Guest are viewing this topic.

Offline RickNY

  • Contributor
  • ***
  • Posts: 104
    • College Hills Weather - Farmingville, NY
Re: Bad analysis data / WXQC Mailing List?
« Reply #25 on: December 14, 2018, 02:51:40 PM »
The NOAA/NWS Meteorological Assimilation Data Ingest System (MADIS) is NOT just for local wx, it is *THE* data collecting (and analysis) system that feeds the WHOLE USA weather mapping program. We just "see" it's local results based upon whatever stations it has selected as its "pool", which changes as the inputs change; it's NOT always the same stations being sampled and used.

Some common "input" problems that MADIS contends with are: different timestamps, units, wrong elevations, incorrect locations (lat & long), etc.

I think it would be reasonable to expect that the whole system would reject as invalid dewpoints of 20,000 degrees.. And not flag a slew of data as not meeting QC because it was 19,950 degrees low.. Just saying..

Offline Old Tele man

  • Singing in the rain...
  • Forecaster
  • *****
  • Posts: 1389
Re: Bad analysis data / WXQC Mailing List?
« Reply #26 on: December 14, 2018, 03:21:12 PM »
The NOAA/NWS Meteorological Assimilation Data Ingest System (MADIS) is NOT just for local wx, it is *THE* data collecting (and analysis) system that feeds the WHOLE USA weather mapping program. We just "see" it's local results based upon whatever stations it has selected as its "pool", which changes as the inputs change; it's NOT always the same stations being sampled and used.

Some common "input" problems that MADIS contends with are: different timestamps, units, wrong elevations, incorrect locations (lat & long), etc.

I think it would be reasonable to expect that the whole system would reject as invalid dewpoints of 20,000 degrees.. And not flag a slew of data as not meeting QC because it was 19,950 degrees low.. Just saying..
Computer programs use algorithms which sometimes -- even though they shouldn't -- dumbly try to "divide-by-Zero" or slowly "round-off" results to zero, which BLOWS UP the analysis, producing GIGO.
SYS: Davis VP2 Vue/WL-IP & Envoy8X/WL-USB;
DBX2 & DBX1 Precision Digital Barographs
CWOP: DW6988 - 2 miles NNE of Cortaro, AZ
WU - KAZTUCSO202, Countryside

Offline galfert

  • Forecaster
  • *****
  • Posts: 1218
Re: Bad analysis data / WXQC Mailing List?
« Reply #27 on: December 14, 2018, 04:21:45 PM »
Old Tele man,
This is not a MADIS issue that RickNY is reporting and as others have seen. You can look at the MADIS analysis and the results are fine. The issue being reported only has to do with Gladstonefamily.net. That is a poorly run system that ingests MADIS data and then does its own separate analysis. That is where the problem is ....not MADIS.

But to your credit you are correct that there are times when MADIS does ingest bad data from bad stations. But never have I seen MADIS report thousands of degrees of error.  MADIS is very good at throwing out really far off bad data. MADIS does make minor errors as you are reporting though. Unfortunately Gladstonefamily hick-ups or something with its own far off data analysis nationwide for no apparent reason sometimes.
WS-2000 & WS-2902A | ObserverIP | WeatherBridge (Meteobridge)
WU: KFLWINTE111  |  PWSweather: KFLWINTE111
CWOP: FW3708  |  AWEKAS: 14814
Tele-Pole flag pole is here (not installed yet)

Offline Old Tele man

  • Singing in the rain...
  • Forecaster
  • *****
  • Posts: 1389
Re: Bad analysis data / WXQC Mailing List?
« Reply #28 on: December 14, 2018, 04:36:48 PM »
Old Tele man,
This is not a MADIS issue that RickNY is reporting and as others have seen. You can look at the MADIS analysis and the results are fine. The issue being reported only has to do with Gladstonefamily.net. That is a poorly run system that ingests MADIS data and then does its own separate analysis. That is where the problem is ....not MADIS.

But to your credit you are correct that there are times when MADIS does ingest bad data from bad stations. But never have I seen MADIS report thousands of degrees of error.  MADIS is very good at throwing out really far off bad data. MADIS does make minor errors as you are reporting though. Unfortunately Gladstonefamily hick-ups or something with its own far off data analysis nationwide for no apparent reason sometimes.

Agreed, the RED X's and GREEN check marks √ are assigned by CWOP/Gladstone, but they're assigned to data coming from MADIS and errors in MADIS are propagated into CWOP/Gladstone; and, Gladstone software can blow-up too, although mostly I've seen time-stamp errors shifting the red and blue traces further and further apart horizontally (thus being TIME and NOT data error) until eventually hiccups or total drop outs occur.
« Last Edit: December 14, 2018, 04:39:34 PM by Old Tele man »
SYS: Davis VP2 Vue/WL-IP & Envoy8X/WL-USB;
DBX2 & DBX1 Precision Digital Barographs
CWOP: DW6988 - 2 miles NNE of Cortaro, AZ
WU - KAZTUCSO202, Countryside

 

anything