Author Topic: State of weather software, networks and contribution "quality"  (Read 4189 times)

0 Members and 1 Guest are viewing this topic.

Offline boss281

  • Senior Contributor
  • ****
  • Posts: 198
This may need to be moved, but I had to start software.

I'm a rock bottom amateur.  My first weather station from Ambient was picked up in in the late 2000s after moving to an area that appeared to have abnormally high wind during the winter time, compared to previous places I've lived.  The station seemed to confirm that, and I joined the WU to contribute.  That was about the depth of my interest, and the station worked well.

It died a few months ago, and I replaced it with a more expensive Davis VP2 connected to a data logger and Windows laptop full time.  I used the Davis WeatherLink software for a bit, dabbled with internet based push vs Windows based push, and after hearing the Win version was no longer getting updates, searched around a bit and have (temporarily) landed on Cumulus, and push out to WU, CWOP and PWS.

Of course, Cumulus support from the developer ended some time ago, so even with great support from that community, eventually I switch again.  I'm also a bit troubled that CWOP itself seems aged (the website hasn't been updated in quite some time it appears).

Where are the progressive internet based communities?  Who is developing software and is expected to do so into the future?  How is weather data being scrubbed for QC, and using what protocols?  From a rock bottom amateur lens, this seems a bit in disarray.  Am I wrong?  Have I not looked into this deep enough?

I just want to get good software with good support (for Win 7 or 10 machines) that supports the hardware, join the right networks so I can contribute to the community, and get feedback about data so I can attempt to correct if possible (I swear the wind data is reading too low out of this new hardware).

Thanks, just feeling a bit blue about it all...
Weather Underground PWS KMDNEWWI3

Online johnd

  • Forecaster
  • *****
  • Posts: 4827
    • www.weatherstations.co.uk
Re: State of weather software, networks and contribution "quality"
« Reply #1 on: April 18, 2018, 09:24:11 AM »
OK, let me have a first stab - it's an interesting question.

To start with just a couple of quick corrections: Weatherlink software development has not stopped at all, it's just moved mostly online, ie with weatherlink.com; it's become a cloud application or platform, as is happening with a lot of other software. And there will still likely be both minor updates and continuing support for local WL - it's major new features that are no longer expected.

And CumulusMX development hasn't stopped AFAIK, it was paused while the author took an extended break and had other priority concerns. But I think there was a recent post from Steve implying that he was going to look at resuming some work on Cumulus - you'd need to check the CMX forum for details.

But on the larger question of WX software development: To be brutal, too many people want something for nothing, which just can't work indefinitely, at least not without dedicated volunteers with good coding skills and some tight central direction with a clear roadmap for the application. I guess weewx is the closest to an open-source wx software community and that seems to work pretty well, albeit needing some basic skills to set up the system so maybe not the ideal option for those without an affinity for computers. If you want an online platform and willing to pay for extra features then weatherlink.com is maybe the best candidate there.

Otherwise, whether free or paid-for (and with WU as a possible exception), wx software tends to be developed by single individuals and you're then obviously locked in to their continuing interest, health, other calls on their time, their skill-set, their vision for the product etc. I suspect that the real killer is providing support - once you get to eg 1000-2000 users and growing then, especially with novice users and non-native English speakers (or whatever language the author speaks) then support can become a full-time job.

Potentially if you could find eg 10,000 users each willing to subscribe $20 yearly then over a year or two a good new wx platform could be developed. But how do you get that commitment off the ground and then develop the coordination?

Edit: I guess what's needed is something like Kickstarter or Indiegogo but AFAICS aren't these mostly for hardware rather than software projects?

QC is a great idea in my book, but I don't underestimate the challenges in achieving it.
« Last Edit: April 18, 2018, 01:18:16 PM by johnd »
Prodata Weather Systems
Prodata's FAQ/support site for Davis stations
Includes many details on 6313 Weatherlink console.
UK Davis Premier Dealer - All Davis stations, accessories and spares
Cambridge UK

Sorry, but I don't usually have time to help with individual issues by email unless you are a Prodata customer. Please post your issue in the relevant forum section here & I will comment there if I have anything useful to add.

Offline WheatonRon

  • Forecaster
  • *****
  • Posts: 1237
    • WUnderground
Re: State of weather software, networks and contribution "quality"
« Reply #2 on: April 21, 2018, 10:25:30 PM »
This may need to be moved, but I had to start software.

...


Of course, Cumulus support from the developer ended some time ago, so even with great support from that community, eventually I switch again.  I'm also a bit troubled that CWOP itself seems aged (the website hasn't been updated in quite some time it appears).

...

Thanks, just feeling a bit blue about it all...

Your comment on CWOP being out of date is right on target. Since Russ Chadwick at NOAA in Boulder, Colorado retired a few years back, the CWOP website has grown terribly out of date. I, and several other amateur weather enthusiasts, have offered to help (free volunteer help no less) update CWOP by identifying specific content that is out-of-date and how to fix, but such offers have fallen on deaf ears. Someone from NOAA or its equivalent, should dedicate some time and effort that Russ provided. Until this happens, or volunteer help is used or at least considered, CWOP will continue to drift in looking and being current.

For your other comments, I too am concerned Davis is dropping further updates to its WL software. The software is clunky looking (think Windows 95) but it provides the historical data I want and need.
« Last Edit: April 21, 2018, 11:19:36 PM by WheatonRon »
Davis VP2 with SHT31 (3 complete VP2 systems—2 with a daytime fan and 1 that has a 24 hour fan); CWOP--CW5020, FW3075 and FW4350; WU--KILWHEAT17, KILWHEAT36 and KILWHEAT39; WeatherCloud.net; CoCoRaHS--IL-DP-132; and Weatherlink 2.0

Offline doubleohwhatever

  • Senior Contributor
  • ****
  • Posts: 252
Re: State of weather software, networks and contribution "quality"
« Reply #3 on: April 22, 2018, 02:07:23 AM »
Otherwise, whether free or paid-for (and with WU as a possible exception), wx software tends to be developed by single individuals and you're then obviously locked in to their continuing interest, health, other calls on their time, their skill-set, their vision for the product etc.
Unfortunately this is always going to be a problem unless the software is being developed by a larger company or everything is open source. At the moment, I'm using MeteoBridge and there's always the chance that the next software update will be the last.

What I would love to see is an open source project based on Raspberry Pi that includes a modern UI and a simple/cheap RF hat (again, open source) to directly receive data from the various stations.

The reasons I think this would be the best hardware approach:

1) A Raspberry Pi is easily obtainable for a lot of people all over the world

2) With an open source RF hat, anyone or any company could produce them.

Basically, we wouldn't have to worry about a single company having a monopoly on a the hardware yet anyone or any company could still sell complete/assembled kits for profit.

As for the software, we need something like WeeWX but with some improvements:

1) Disk images available - Basically make setup as easy as writing the image to an SD card and tossing it in a Raspberry Pi.

2) All configuration done via web UI with an optional setup wizard

3) Modern web UI

Honestly, all of that could most likely be built on top of WeeWX.

Edit: I guess what's needed is something like Kickstarter or Indiegogo but AFAICS aren't these mostly for hardware rather than software projects?
Software is allowed and I'd definitely support a project that hits on at least some of what I described above.

Online johnd

  • Forecaster
  • *****
  • Posts: 4827
    • www.weatherstations.co.uk
Re: State of weather software, networks and contribution "quality"
« Reply #4 on: April 22, 2018, 04:28:36 AM »
Software is allowed and I'd definitely support a project that hits on at least some of what I described above.

I did look into this slightly more and the impression I got was that software projects are not encouraged under Kickstarter, but that Indiegogo is more flexible.

On the software front, someone has to get the ball rolling by writing a reasonably detailed draft specification for the software. Until this is done and also until debated, modified and finally agreed by some core project team then I'm not sure how it's going to progress.

For instance I can see a lot of potential debate between those who'd prefer a fully local solution and those who would rather have everything possible in the cloud (not that the two are totally exclusive but trying to ride more than one horse to start with will be tricky). And another battlefront will be between those in the simple, clean, 'less is more' camp (count me in) and those who press for more and more readings of every conceivable kind to be included (resulting in what to me is the bane of a lot of wx software - just too much information crammed into the display space ( typically far to much for the layman to take in at a glance) and often not very elegantly presented).

Another major question is how units should be handled - personally I'd stick with just one system of units throughout until the final display stage when appoprriate conversions can be made.

And one other difficulty is how to actually recruit coders to the project with good technical skills in the various required areas.

One way forwards might be to break the project down into separate functional areas, with agreed interfaces between them. My picture has always been:

1. Data acquisition + initial processing/summary (definitely Pi territory)
2. Uploads of real-time data to eg an MQTT broker in an agreed extensible format eg JSON-based (Pi)
3. Long-term storage on a reliable platform (probably not Pi because of the lack of robust memory - better suited to cloud)
4. Real-time browser-based front-end receiving MQTT messages pushed from the broker
5. Additional functionality in the browser-based front-end to provide reporting and retrospective charting via queries (probably via websockets) to the cloud database.

We can debate the details, but the point is to break things into more manageable chunks with agreed interfaces between them.
Prodata Weather Systems
Prodata's FAQ/support site for Davis stations
Includes many details on 6313 Weatherlink console.
UK Davis Premier Dealer - All Davis stations, accessories and spares
Cambridge UK

Sorry, but I don't usually have time to help with individual issues by email unless you are a Prodata customer. Please post your issue in the relevant forum section here & I will comment there if I have anything useful to add.

Offline captgadget

  • Forecaster
  • *****
  • Posts: 359
Re: State of weather software, networks and contribution "quality"
« Reply #5 on: May 11, 2018, 09:15:23 AM »
Wow! A very interesting discussion. Boy as much as love all the weather sites I visit I'm really grateful for those who have all the skills and talents to put those together. I sure wish I could help on a new venture but I have not coding skills. I'm the guy who puts the key in the ignition switch and drives off. I certainly hope something good can come of all this. Quite interesting to hear the comments on CWOP. As far as I know isn't that how mesowest gets their information? Good luck and if there is something I could do let me know but I know no coding.

Offline doubleohwhatever

  • Senior Contributor
  • ****
  • Posts: 252
Re: State of weather software, networks and contribution "quality"
« Reply #6 on: May 11, 2018, 12:19:10 PM »
I gave up trying to get my station registered with CWOP. If they can't reply to an email they obviously don't want my data.

Are there any weather networks that actually *want* PWS data these days? It feels like wunderground couldn't care less at the moment. I'd really like to join a network that actually wants PWS data.

Offline PaulMy

  • Forecaster
  • *****
  • Posts: 5509
    • KomokaWeather
Re: State of weather software, networks and contribution "quality"
« Reply #7 on: May 11, 2018, 12:46:36 PM »
Not sure about "want" but Weathercloud, PWS, AWEKAS and WOW are other possibilities!

Enjoy,
Paul

Offline WheatonRon

  • Forecaster
  • *****
  • Posts: 1237
    • WUnderground
Re: State of weather software, networks and contribution "quality"
« Reply #8 on: May 11, 2018, 12:56:39 PM »
I gave up trying to get my station registered with CWOP. If they can't reply to an email they obviously don't want my data.

Are there any weather networks that actually *want* PWS data these days? It feels like wunderground couldn't care less at the moment. I'd really like to join a network that actually wants PWS data.

See my prior post in this thread re CWOP—very sorry as it used to be user friendly—no more, unfortunately. It still accepts data and it is very useful and provides data to MesoWest at the University of Utah, but getting support from CWOP is next to impossible.

Last week, I started sending my PWS data to Weathercloud.net. I don’t know anything about them, good or bad, but they at least respond to emails. When signing up, I was using Davis’ Weatherlink software (version 6.0.3) that I use to upload to CWOP and WU. When I added weathercloud, it removed my WU settings! I added the WU settings back into WeatherLink and now all three uploads work. I alerted weathercloud of this issue and at least they acknowledged my email. Whether they do anything, who knows.
« Last Edit: May 11, 2018, 01:53:54 PM by WheatonRon »
Davis VP2 with SHT31 (3 complete VP2 systems—2 with a daytime fan and 1 that has a 24 hour fan); CWOP--CW5020, FW3075 and FW4350; WU--KILWHEAT17, KILWHEAT36 and KILWHEAT39; WeatherCloud.net; CoCoRaHS--IL-DP-132; and Weatherlink 2.0

Offline captgadget

  • Forecaster
  • *****
  • Posts: 359
Re: State of weather software, networks and contribution "quality"
« Reply #9 on: May 11, 2018, 03:34:59 PM »
Are there any weather networks that actually *want* PWS data these days? It feels like wunderground couldn't care less at the moment. I'd really like to join a network that actually wants PWS data.

I send my weather to pwsweather.com, and you can also check the weather at weatherforyou.com and if you want he will connect your pws to his webpage. I corresponded with him by email and he's very easy to work with. Highly recommended.

Offline WheatonRon

  • Forecaster
  • *****
  • Posts: 1237
    • WUnderground
Re: State of weather software, networks and contribution "quality"
« Reply #10 on: May 11, 2018, 03:58:06 PM »

I send my weather to pwsweather.com, and you can also check the weather at weatherforyou.com and if you want he will connect your pws to his webpage. I corresponded with him by email and he's very easy to work with. Highly recommended.

Unfortunately, Davis’ Weatherlink doesn’t enable PWS uploads to pwsweather.com, so one must use a different software.
Davis VP2 with SHT31 (3 complete VP2 systems—2 with a daytime fan and 1 that has a 24 hour fan); CWOP--CW5020, FW3075 and FW4350; WU--KILWHEAT17, KILWHEAT36 and KILWHEAT39; WeatherCloud.net; CoCoRaHS--IL-DP-132; and Weatherlink 2.0

Online johnd

  • Forecaster
  • *****
  • Posts: 4827
    • www.weatherstations.co.uk
Re: State of weather software, networks and contribution "quality"
« Reply #11 on: May 11, 2018, 03:59:00 PM »
Last week, I started sending my PWS data to Weathercloud.net. I don’t know anything about them, good or bad, but they at least respond to emails.

Isn't it likely though that this is just a question of scale and long-term commitment. If you're trying to establish a new network and still keen and are getting maybe 10-20 emails/day then responding is perfectly practicable. 3-4 years down the track, the network has grown substantially; you're still doing it on a totally voluntary basis though the enthusiasm is starting to wane but you're now getting eg 200 emails/day and it's somewhere between unappealing and unfeasible to answer every email.

Not saying this is right or wrong but I can understand the problem as networks grow. The only answer long-term that I can see is to commercialise it somehow, either as a free service that sells advertising and collated data or as a subscription service where you pay a modest amount (eg $10-20pa) or some mix of both. That would pay at least some remuneration to the 'volunteers' and also the hardware cost or cloud fees for hosting the network.
Prodata Weather Systems
Prodata's FAQ/support site for Davis stations
Includes many details on 6313 Weatherlink console.
UK Davis Premier Dealer - All Davis stations, accessories and spares
Cambridge UK

Sorry, but I don't usually have time to help with individual issues by email unless you are a Prodata customer. Please post your issue in the relevant forum section here & I will comment there if I have anything useful to add.

Offline WheatonRon

  • Forecaster
  • *****
  • Posts: 1237
    • WUnderground
Re: State of weather software, networks and contribution "quality"
« Reply #12 on: May 11, 2018, 04:04:28 PM »
Last week, I started sending my PWS data to Weathercloud.net. I don’t know anything about them, good or bad, but they at least respond to emails.

Isn't it likely though that this is just a question of scale and long-term commitment. If you're trying to establish a new network and still keen and are getting maybe 10-20 emails/day then responding is perfectly practicable. 3-4 years down the track, the network has grown substantially; you're still doing it on a totally voluntary basis though the enthusiasm is starting to wane but you're now getting eg 200 emails/day and it's somewhere between unappealing and unfeasible to answer every email.

Not saying this is right or wrong but I can understand the problem as networks grow. The only answer long-term that I can see is to commercialise it somehow, either as a free service that sells advertising and collated data or as a subscription service where you pay a modest amount (eg $10-20pa) or some mix of both. That would pay at least some remuneration to the 'volunteers' and also the hardware cost or cloud fees for hosting the network.

You are 100% correct. I certainly don’t want the government to get involved, but I would certainly pay a reasonable sum on an annual basis to have consistent, high quality, PWS data available to me as well as accepting my PWS uploads and those of others that have high-quality PWS gear.
« Last Edit: May 11, 2018, 04:09:07 PM by WheatonRon »
Davis VP2 with SHT31 (3 complete VP2 systems—2 with a daytime fan and 1 that has a 24 hour fan); CWOP--CW5020, FW3075 and FW4350; WU--KILWHEAT17, KILWHEAT36 and KILWHEAT39; WeatherCloud.net; CoCoRaHS--IL-DP-132; and Weatherlink 2.0

Offline SLOweather

  • Global Moderator
  • Forecaster
  • *****
  • Posts: 3456
    • Weatherelement Moline IL
Re: State of weather software, networks and contribution "quality"
« Reply #13 on: May 11, 2018, 06:37:02 PM »
...I would certainly pay a reasonable sum on an annual basis to have consistent, high quality, PWS data available to me as well as accepting my PWS uploads and those of others that have high-quality PWS gear.

What's "reasonable"? What does "available to (you)" look like?

Offline WheatonRon

  • Forecaster
  • *****
  • Posts: 1237
    • WUnderground
Re: State of weather software, networks and contribution "quality"
« Reply #14 on: May 11, 2018, 08:03:57 PM »
...I would certainly pay a reasonable sum on an annual basis to have consistent, high quality, PWS data available to me as well as accepting my PWS uploads and those of others that have high-quality PWS gear.

What's "reasonable"? What does "available to (you)" look like?

$20 to $50 per year. Only high quality PWS stations--those that receive passing grades by MesoWest would be my criteria.
Davis VP2 with SHT31 (3 complete VP2 systems—2 with a daytime fan and 1 that has a 24 hour fan); CWOP--CW5020, FW3075 and FW4350; WU--KILWHEAT17, KILWHEAT36 and KILWHEAT39; WeatherCloud.net; CoCoRaHS--IL-DP-132; and Weatherlink 2.0

Offline jas340

  • Senior Member
  • **
  • Posts: 92
    • Lone Pine Ranch  Horseshoe Bend, Idaho  USA
Re: State of weather software, networks and contribution "quality"
« Reply #15 on: May 12, 2018, 08:46:05 PM »
I remember reading on the PWS weather site about 6-8 weeks ago that they were shutting down in mid-May of this year. I just went over there since I see PWS weather mentioned in this thread and I see no info on the shutdown. Obviously they changed their minds for some reason or other.

Offline waiukuweather

  • Forecaster
  • *****
  • Posts: 1072
Re: State of weather software, networks and contribution "quality"
« Reply #16 on: May 12, 2018, 11:32:05 PM »
Weather Bug is shutting down
maybe you are confused with that one?

Offline WheatonRon

  • Forecaster
  • *****
  • Posts: 1237
    • WUnderground
Re: State of weather software, networks and contribution "quality"
« Reply #17 on: May 13, 2018, 09:26:18 AM »
Weather Bug is shutting down
maybe you are confused with that one?

I hadn’t heard that about WeatherBug. Where did you see or hear that?
Davis VP2 with SHT31 (3 complete VP2 systems—2 with a daytime fan and 1 that has a 24 hour fan); CWOP--CW5020, FW3075 and FW4350; WU--KILWHEAT17, KILWHEAT36 and KILWHEAT39; WeatherCloud.net; CoCoRaHS--IL-DP-132; and Weatherlink 2.0

Offline waiukuweather

  • Forecaster
  • *****
  • Posts: 1072
Re: State of weather software, networks and contribution "quality"
« Reply #18 on: May 13, 2018, 02:16:51 PM »
Nathan made an announcement here on wxforum

Offline dsscheibe

  • Senior Member
  • **
  • Posts: 98
    • Scott's Weather Station Cactus Wren, Phoenix, AZ
Re: State of weather software, networks and contribution "quality"
« Reply #19 on: September 18, 2018, 04:30:02 AM »
I switched to Weather Display from Virtual Weather station because the latter kept crashing in Win10 and Ambient doesn't seem interested in fixing it.  Weather Display has a definite learning curve and isn't what I'd call easy to use. How ever it supports about anything, will send data to every place I've heard of and many I had not.  The author is active on the https://www.weather-watch.com/smf/index.php form and will answer questions eventually.
Scott
Davis Pro2 Plus daytime aspirated, wired
Weather Display