Author Topic: Time to quit sending data  (Read 6272 times)

0 Members and 1 Guest are viewing this topic.

Offline Scalphunter

  • Forecaster
  • *****
  • Posts: 2314
Time to quit sending data
« on: January 07, 2017, 11:51:48 PM »
 Looks to me this one size fits all software for sort out the good data from the bad is for the birds.  First madis  and  CWOP  tells one their data/equipment is bad  now have MesoWest flagging  data as bad. Time for them to pull their heads out of their butts and  l;ook at what is really going on.  Station  higher then the ones in  the bottom of drainage are going to have higher temps  in the winter. Wonder if these highly educated types ever or even heard of inversion. It is always the PWS stations that are flagged I have yet to see an government  site flagged even when you know the data is bad. Not sure about you  but I know my data good and I am aware of what happens in winters here and you can  go along the road and see temperatures  differ as much as 20-25 degrees.  So can't tell me  these so call smart ones can't figure it out. Can save an bundle  just keeping the data here on my laptop and get rid of the internet hot spot.


John

Offline CW2274

  • Forecaster
  • *****
  • Posts: 6731
    • Conditions @ CW2274 West Tucson-Painted Hills Ranch
Re: Time to quit sending data
« Reply #1 on: January 08, 2017, 12:01:09 AM »
Have you tried telling them? I've spoken to MW before and had excellent results.

Offline Scalphunter

  • Forecaster
  • *****
  • Posts: 2314
Re: Time to quit sending data
« Reply #2 on: January 08, 2017, 12:18:19 AM »
Yes I have . It d0ose  nothing but waste my time.  Every winter it the same thing.

  John

Offline ocala

  • Forecaster
  • *****
  • Posts: 4383
Re: Time to quit sending data
« Reply #3 on: January 08, 2017, 06:50:46 AM »
Problem is there's no way to check every station for accuracy so this is the system they use.
It's not perfect but as you said you know your data is good.
That's all that matters.
The blues had a baby and they named it Rock & Roll

Offline Jáchym

  • Meteotemplate Developer
  • Forecaster
  • *****
  • Posts: 8605
    • Meteotemplate
Re: Time to quit sending data
« Reply #4 on: January 08, 2017, 08:31:52 AM »
John, when I read your post it was as if I wrote myself what happened to me about 2 yrs ago... exact same thing. I got an email from CWOP saying my station is showing too high values (that was back when the station was positioned much better than now, now I know it is not accurate, but back then it was). They also told me that they disabled my temp sensor for 24h and I should fix it.
I replied to them saying that I live right in the city center, so obviously the temp is higher than the one on the airport outside the city. The guy on the other end was very arrogant and only replied that my station is showing nonsense and that if I argue and dont do anything about it, they would disable my temp sensor on their site permanently.
When I got that reply, I just thought to myself - seriously? Do they think I am the one dependent on them? I just replied to him to please disable my entire station that I no longer wish to send them anything because I absolutely dont care and it is only going to put less load on my router.

And slightly unrelated.... about a yr ago, I wanted to do a plugin for my template that would allow users to upload data to CWOP from my template - I have the same thing for WU and one user asked for it. I only needed some info about one of the things in the upload protocol. Tried contacting them 3 times (and I want to emphasize that I contacted them from a different email than I used for my station - just in case they remembered and it was the same guy on the other hand).
I never got any reply, not even negative. Nothing.

Thats my experience with CWOP... and based on the experience I have with WU you probably understand why Im not a big fan of all these community networks that are in fact commercial and why I created my template, where I have my data saved on my server and no-one telling me what it should look like.

Offline weatherc

  • Senior Contributor
  • ****
  • Posts: 278
Re: Time to quit sending data
« Reply #5 on: January 08, 2017, 09:16:37 AM »
My 2 €urocents as developer of EWN...

I have for long time said any comparement to close stations nor even to previous data from same station as just a big fail and show in some way how un-proffessional the network devs are. The weather, and how it changes, are not same in ie. Paris as in north Scandinavia.

My findings with EWN:
- The ONLY case where a compare to previous data work are for BIG changes like sensor-fail, ie. a drop from +20 to -20. For example, here in Finland did we a couple of days ago see a 10C rise in temperature in just 10 minutes at one of the official Finnish met. institute stations
- Comparement to nearbour stations are totally useless. Just look at ie. Norway with its high mountains and deep fjords. It can be <0C in the mountains while it is +15C at the coast with just some km distance.

That said, there are still a problem, solar-heated stations and how to detect them. Compare to previous data should in some way work with a ie. "+-2° in 15 minutes"-setting but setting its limit too tight would trigger error for above example. One solution are of course to not allow any stations without a proper shield but its not a good solution either as there can be even "cheapy stations like Netamo" with proper readings.

Offline Jáchym

  • Meteotemplate Developer
  • Forecaster
  • *****
  • Posts: 8605
    • Meteotemplate
Re: Time to quit sending data
« Reply #6 on: January 08, 2017, 09:41:50 AM »
100% agreed

When we work at a pro level here at work, we never base comparison on position. Well actually we do, but what is important probably even more is elevation. And even that is not enough, other factors include urban/rural station, exposure to wind (depends on the site of the station, if it is in an open field on an airport then obviously the wind for example is going to be different compared to a pro station in the city center build-up zone).

You are absolutely right, they should more look at spikes in the data within that one station, if it was 20° and next minute 5, then something is wrong.

I talked about this before - for example this trend of more and more people buying NetAtmo is dangerous if data validation is done this way. Think of this scenario. You have a very high-quality and well sited accurate station. Your neighbor who has no clue about weather, buys a NetAtmo (or a different sort of these rather design gadgets) and starts sending data to WU as well. Then your other neighbor does the same thing. They both see it more like fun, have it somewhere close to their heated outside walls of the house, exposed to Sun... a cool gadget in their home. Now WU receives data from 3 stations very close to each other - yours and the two NetAtmos. The two NetAtmos are likely to show similar nonsense. Now... if they use the algorithm they use (comparison with neighboring stations), which station´s data do you think will be evaluated as inaccurate... and there is absolutely nothing you can do about it... unless you want to buy additional 3 accurate stations and place them all in your garden to override the others :D

Online Otis

  • (aka Paul)
  • Forecaster
  • *****
  • Posts: 724
    • Lake Huron Weather
Re: Time to quit sending data
« Reply #7 on: January 08, 2017, 09:49:25 AM »
Simple solution for those that are unhappy with CWOP - quit sending your data.

CW3699

Offline weatherc

  • Senior Contributor
  • ****
  • Posts: 278
Re: Time to quit sending data
« Reply #8 on: January 08, 2017, 10:43:01 AM »
Simple solution for those that are unhappy with CWOP - quit sending your data.

Its not about stop sending data, its about that those network devs think the nonsense-datas are the accurate ones and the accurate ones are in-accurate only beacuse there are more nonsense-stations neaby than the only accurate one just like Jáchym's above example.
« Last Edit: January 08, 2017, 10:44:48 AM by weatherc »

Offline hankster

  • Forecaster
  • *****
  • Posts: 482
    • NFM Weather
Re: Time to quit sending data
« Reply #9 on: January 08, 2017, 10:46:34 AM »
Simple solution for those that are unhappy with CWOP - quit sending your data.

Yeah, that's it. Like cutting off your nose to spite your face.

Online Otis

  • (aka Paul)
  • Forecaster
  • *****
  • Posts: 724
    • Lake Huron Weather
Re: Time to quit sending data
« Reply #10 on: January 08, 2017, 10:54:29 AM »
I'm happy with CWOP - for those that are not - quit complaining about them and stop sending your data.  Simple solution.

CW3699

Offline Jáchym

  • Meteotemplate Developer
  • Forecaster
  • *****
  • Posts: 8605
    • Meteotemplate
Re: Time to quit sending data
« Reply #11 on: January 08, 2017, 11:09:10 AM »
No, its not a simple solution. It is a solution for me, but you have to remember that there are majority of people who might look at CWOP, not sending data to them, or being unaware of this and they are mistified! They are shown data that is not accurate because of the poorly designed algorithm CWOP uses (and WU and others) to determine data validity. If you described their system to me or any of my colleagues or other professionals in the field, they will laugh at it.
We spend hours validating our data and if it was this easy it would be great, unfortunately its not.

Online Otis

  • (aka Paul)
  • Forecaster
  • *****
  • Posts: 724
    • Lake Huron Weather
Re: Time to quit sending data
« Reply #12 on: January 08, 2017, 11:49:03 AM »
Complain about WU.
Complain about CWOP.
So what is the point of the complaints?
Lead, follow, or get out of the way - applies to a lot of aspects in life.

CW3699

Offline Jáchym

  • Meteotemplate Developer
  • Forecaster
  • *****
  • Posts: 8605
    • Meteotemplate
Re: Time to quit sending data
« Reply #13 on: January 08, 2017, 12:03:06 PM »
Right so if for example they broadcast nonsense on TV, or write misleading info in the news, you are going to say "Im not going to read it and therefore it is ok.".

If thats the way you look at it, then yes, I guess you are right with the WU, CWOP issue.

it does not bother me as such, it bothers me because it misleads others and uses some highly amateur algorithm and pretends to be a reliable source

Online Otis

  • (aka Paul)
  • Forecaster
  • *****
  • Posts: 724
    • Lake Huron Weather
Re: Time to quit sending data
« Reply #14 on: January 08, 2017, 12:11:16 PM »
To clarify
I never said I have a problem with either WU or CWOP.
Please don't put words in my mouth.

CW3699

Offline Scalphunter

  • Forecaster
  • *****
  • Posts: 2314
Re: Time to quit sending data
« Reply #15 on: January 08, 2017, 05:38:24 PM »
Otis  some thing tells me that back in your flatlands where the trash pile is the tallest  mound around  you are  just go with the flow type.  Just because You have no problem with  WU,CWOP and what ever  does it mean that it is right  and ok.   I guess that  a station  in Alaska  can be compared to an station in Floridia and  that the  Flordian station is ok cause it fits the profile in data  while the station in Alaska is wrong cause it falls out side the parameters. That the way I am seeing your point of  view.  I been around  lot longer  then you and just come in from cutting up an cord  and half of wood while  had daylight to do.  yep we do lead here in fact  when the going get tough we are just getting started.
  madis  and CWOP are going to have to come to grips some day that the software they use doesn't fit all.   L3 test is if your station is more then 15 deg  then other stations around you it an fail.  L2 is even more hooky.  One thing would  be an help if they tell you what their reference  station  are  but no they are top secret and like any other government  enity  they keep you in the dark and  feed you BS. Maybe they should  just stop taking PWS data  only.   
   Speaking of Netatomes station we have one on WU  that at 30 below it was saying 80  deg F. yet it passing the  data check on WU.  So as you can see it bad data  but according to you it should be uses cause they like to show their data even if it is bad.
  Back to cutting  more wood while got  hour of daylight left. Balmy day out  from -16 when I started it is -8 out now.


John


 


Offline Jáchym

  • Meteotemplate Developer
  • Forecaster
  • *****
  • Posts: 8605
    • Meteotemplate
Re: Time to quit sending data
« Reply #16 on: January 08, 2017, 05:50:45 PM »
Otis  some thing tells me that back in your flatlands where the trash pile is the tallest  mound around  you are  just go with the flow type.  Just because You have no problem with  WU,CWOP and what ever  does it mean that it is right  and ok.   I guess that  a station  in Alaska  can be compared to an station in Floridia and  that the  Flordian station is ok cause it fits the profile in data  while the station in Alaska is wrong cause it falls out side the parameters. That the way I am seeing your point of  view.  I been around  lot longer  then you and just come in from cutting up an cord  and half of wood while  had daylight to do.  yep we do lead here in fact  when the going get tough we are just getting started.
  madis  and CWOP are going to have to come to grips some day that the software they use doesn't fit all.   L3 test is if your station is more then 15 deg  then other stations around you it an fail.  L2 is even more hooky.  One thing would  be an help if they tell you what their reference  station  are  but no they are top secret and like any other government  enity  they keep you in the dark and  feed you BS. Maybe they should  just stop taking PWS data  only.   
   Speaking of Netatomes station we have one on WU  that at 30 below it was saying 80  deg F. yet it passing the  data check on WU.  So as you can see it bad data  but according to you it should be uses cause they like to show their data even if it is bad.
  Back to cutting  more wood while got  hour of daylight left. Balmy day out  from -16 when I started it is -8 out now.


John

Hi John,
just curious, does that NetAtmo PWS that showed 110deg more than it should have the prestigious gold star? :D

Offline ocala

  • Forecaster
  • *****
  • Posts: 4383
Re: Time to quit sending data
« Reply #17 on: January 08, 2017, 07:15:50 PM »
John they do tell you what stations you are being compared with.
On the Daily Weather Quality Charts it says this.
These charts compare the reported data from DW1453 with the predicted data for that location for 2017-Jan-08. The predicted data comes from the surrounding stations -- some of which are listed below.
There are 10 stations listed at the bottom of the page. Besides those 10 I don't know how many more they use for analysis.
The blues had a baby and they named it Rock & Roll

Offline Scalphunter

  • Forecaster
  • *****
  • Posts: 2314
Re: Time to quit sending data
« Reply #18 on: January 09, 2017, 12:08:58 AM »
Quote
Note that this is not the list of stations that are used in the data analysis. That list of stations is known only to MADIS]/quote]

 Then this statement made just above what you sent is wrong. 

 There nothing that I can do about it anyways. As  an gussuk implied  I should just get out of the way as other are the leader and  I'm only an complainer.


catch all later

John

Offline SoMDWx

  • Forecaster
  • *****
  • Posts: 1014
    • Southern Maryland Weather
Re: Time to quit sending data
« Reply #19 on: January 09, 2017, 10:00:05 AM »
Right so if for example they broadcast nonsense on TV, or write misleading info in the news, you are going to say "Im not going to read it and therefore it is ok.".

If thats the way you look at it, then yes, I guess you are right with the WU, CWOP issue.

it does not bother me as such, it bothers me because it misleads others and uses some highly amateur algorithm and pretends to be a reliable source

I think I know where Otis is coming from... There is not much anyone can do to control/fix the issues at CWOP, WU, or any other weather data gathering site.

You can chose to send data to whomever you wish and they will do what they want with it.. To get obsessed over incorrect data or any kind of misrepresentation goes over the top.

Free market societies provide a way for others to improve on a given product or service. If you feel that CWOP or others isn't up to par then write your own and publish it....

I provide weather data to many sources. Is it correctly displayed all the time.. No. And with that, I am at peace with that cause the sun will keep coming up everyday, the earth will continue to spin and life will continue to go on...

Jim

Offline BigOkie

  • Forecaster
  • *****
  • Posts: 1367
  • Tulsa, OK
    • KOKTULSA13
Re: Time to quit sending data
« Reply #20 on: January 09, 2017, 10:47:33 AM »
Right so if for example they broadcast nonsense on TV, or write misleading info in the news, you are going to say "Im not going to read it and therefore it is ok.".

If thats the way you look at it, then yes, I guess you are right with the WU, CWOP issue.

it does not bother me as such, it bothers me because it misleads others and uses some highly amateur algorithm and pretends to be a reliable source

I think I know where Otis is coming from... There is not much anyone can do to control/fix the issues at CWOP, WU, or any other weather data gathering site.

You can chose to send data to whomever you wish and they will do what they want with it.. To get obsessed over incorrect data or any kind of misrepresentation goes over the top.

Free market societies provide a way for others to improve on a given product or service. If you feel that CWOP or others isn't up to par then write your own and publish it....

I provide weather data to many sources. Is it correctly displayed all the time.. No. And with that, I am at peace with that cause the sun will keep coming up everyday, the earth will continue to spin and life will continue to go on...

Jim

Pretty much this.

I don't understand why so many people get bent over what MADIS/CWOP/WU says about your data.  If you're sure that your data is valid, then really that is all that matters.

If it were pay sites that were doing this, then I might have a problem with it.  However we aren't paying to send our data to these sites.

My suggestion, and take it however you want.  Enjoy your weather data even if some service says it's invalid.  If they do then chances are at some point they'll drop you.  Complain about that if you wish, but I wouldn't.
Current setup: Davis Vantage Pro 2 Plus Wireless
Weather radios:
Reecom R-1650
Sangean CL-100
Uniden Home Patrol I

Offline DoctorKnow

  • Forecaster
  • *****
  • Posts: 1982
Re: Time to quit sending data
« Reply #21 on: January 09, 2017, 10:59:56 AM »
Quote
Pretty much this.

I don't understand why so many people get bent over what MADIS/CWOP/WU says about your data.  If you're sure that your data is valid, then really that is all that matters.

That is the way I feel as well. When I first joined, I used to try to get my barometer to match the red line, or the prediction of what the other stations say mine should read, but now, I am not going to sweat a few points. I am going to trust my instrument. I am also not going to have a fit about my anemometer being the lowest speed in the county with all these trees everywhere I have to deal with. My thermometer and hygrometer is IMO one of the most accurate of the lot now on it's own.

Offline Jáchym

  • Meteotemplate Developer
  • Forecaster
  • *****
  • Posts: 8605
    • Meteotemplate
Re: Time to quit sending data
« Reply #22 on: January 09, 2017, 11:11:04 AM »
OK, although...

CWOP goals:

Quote
3) to provide feedback to the data contributors so they have the tools to check and improve their data quality.

Quote
The data undergo quality checking and then are distributed to users.
- suggesting to people viewing the site, the data is accurate


Offline DoctorKnow

  • Forecaster
  • *****
  • Posts: 1982
Re: Time to quit sending data
« Reply #23 on: January 09, 2017, 11:22:01 AM »
OK, although...

CWOP goals:

Quote
3) to provide feedback to the data contributors so they have the tools to check and improve their data quality.

Quote
The data undergo quality checking and then are distributed to users.
- suggesting to people viewing the site, the data is accurate



OK.
Can you explain then why every time I adjusted my barometer higher, the surrounding stations kept going higher, and I would have to re adjust? I gave up and will just leave mine alone... Maybe they will stop trying to beat me...

Offline Jáchym

  • Meteotemplate Developer
  • Forecaster
  • *****
  • Posts: 8605
    • Meteotemplate
Re: Time to quit sending data
« Reply #24 on: January 09, 2017, 11:35:47 AM »
Thats exactly the problem, the algorithm they use is completely flawed.

http://www.wxforum.net/index.php?topic=31148.msg310765#msg310765