Author Topic: Data collection for WiFi-only PWS - beta testers wanted!  (Read 5261 times)

0 Members and 1 Guest are viewing this topic.

Offline relko

  • Member
  • *
  • Posts: 30
  • Astrogenic Systems
    • Astrogenic Systems
Data collection for WiFi-only PWS - beta testers wanted!
« on: September 20, 2018, 11:14:52 AM »
Up to 5 beta testers wanted.

This info will be of relevance to you if:
-You operate a WiFi enabled weather station that is actively uploading to WU but has no built-in capability to collect live or historical data other than retrieving it from WU.
-You have a website, your own Internet or LAN web server or a single board computer (Raspberry PI etc.) where you can store your weather station data on. Basic requirement is a web space or server with PHP and SQLite support.
-You have a 5V/1.0+ amps micro-USB type B power supply laying around (an unused smartphone charger will do the job)

You will be beta testing ATMOCOM which is a small form factor, special purpose WiFi device that you connect your PWS to instead of directly connecting it to your 2.4 GHz Internet router. The device will seamlessly link your PWS and Internet router, send all weather data uploads to the original destination (WU) while simultaneously also re-transmit the data to a user configurable target URL (e.g. on your website). On your web space a special companion PHP receiver/parser script stores the replicated data in monthly zero-configuration SQLite archives. While restricted to the WU data protocol during the beta tests, the PHP script can actually be tailored to support any weather network upload protocol.

In theory any WiFi enabled weather station that can upload data to Weather Underground is compatible. For the past months we have been testing with the following PWS models:
FOSHK WH2900 series (a.k.a. Ambient WS-2902 Osprey, Ventus W830, MiSol, ChiliTec, PanTech, Froggit WH3000, <insert your favorite rebranded variant here> etc.)
GARNI 1055 Arcus (a.k.a. Ambient WS-8478 Falcon, Bresser etc.)
FOSHK HP-3500 series (new unreleased model)

 [ You are not allowed to view attachments ]

A live demo dashboard based on a customized version of Home Weather Station template - designed by Brian at weather34.com and used with kind permission - can be seen here: https://atmocom.com/wx34/?units=metric

Note this dashboard is for demo purposes only, not officially supported by weather34.com!

You should be located in either Europe or North America and know your PWS and Internet equipment well. You should also have some spare time to set up the device when you receive it so that it doesn't sit and collect dust for weeks.

If selected you will get:
-ATMOCOM beta device, which you may keep using with tech support included after tests are completed.

Downloadable:
-User manual
-Data receiver/parser PHP script (Weather Underground protocol)
-Data fetching PHP script (outputs JSON or XML)
-2x graphical weather sticker demo PHP scripts (see signature for an example)

You will have to pay for shipping to your location, estimated 10 EUR ($12 USD) tracked. Package will be declared low value so no import taxes should apply outside the European Union where the items are shipped from.

Key features:
-Minimal form factor. The production version will be the size of a matchbox (beta devices are a bit larger than this)
-Simple configuration via Telnet client when device is in Access Point config mode.
-Plug-configure-play; No operating system dependence, no router, network or any other type of hacks required
-Automatic and fail-safe firmware over the air updates
-Replicate skip option 0-240 seconds. For reducing disk space usage the device can be set to transmit data to your private data store less frequently than the PWS default update rate.
-Numerous watchdog functions to ensure PWS and uplink router WiFi connections are working 24/7/365
-On-board combined status/network traffic LED. Traffic LED indications can be enabled/disabled the user
-A single multifunction button
-Powered by commonly available 5V/1A+ power supply with micro-USB type B connector (not included).

If interested please PM me the following:
-Weather station make and model
-WiFi router make, model and your Internet bandwidth
-Weather networks your PWS is uploading to in addition to WU, if any
-Your WU ID or link

If anything is unclear please feel free to ask below and I'll do my best to answer.
« Last Edit: September 20, 2018, 11:17:09 AM by relko »

Offline kbellis

  • Forecaster
  • *****
  • Posts: 361
Re: Data collection for WiFi-only PWS - beta testers wanted!
« Reply #1 on: January 01, 2019, 02:20:08 PM »
Nice looking template; tip of the hat to Brian.

Why is the user required to have their own dedicated web server if the data is destined for CWOP, for example?

Offline relko

  • Member
  • *
  • Posts: 30
  • Astrogenic Systems
    • Astrogenic Systems
Re: Data collection for WiFi-only PWS - beta testers wanted!
« Reply #2 on: January 02, 2019, 03:15:19 AM »
Why is the user required to have their own dedicated web server if the data is destined for CWOP, for example?

This requirement is outdated, there is now a Python script that handles data logging on the local network. A website with PHP support is only required if data is going to be logged on the Internet. So for the CWOP example you would run the Python script alongside a companion script that can read the logged data and send it on to multiple other targets such as CWOP. Or the Python script, which is open source, can be modified to directly re-transmit to other targets. There is no limit really on possible uses but not everything is implemented at this point in the project.

Offline kbellis

  • Forecaster
  • *****
  • Posts: 361
Re: Data collection for WiFi-only PWS - beta testers wanted!
« Reply #3 on: January 02, 2019, 05:36:07 AM »
Is data logging, local or remote, still a requirement?

Logging implies something is running - somewhere - 24/7, right?

The other point I wanted to be clear on is what, if anything, is impacted by ATMOCOM specifically in regard to the WS-2902A's Display Console as configured through AWNET? Will ATMOCOM interfere in the physical operation of the Ambient system?

Lastly, and not meaning to sound discouraging and I hate even to bring it up, but will running ATMOCOM conflict with any licensing agreements?

Offline relko

  • Member
  • *
  • Posts: 30
  • Astrogenic Systems
    • Astrogenic Systems
Re: Data collection for WiFi-only PWS - beta testers wanted!
« Reply #4 on: January 02, 2019, 01:17:16 PM »
Is data logging, local or remote, still a requirement?

Logging implies something is running - somewhere - 24/7, right?

Data logging is not a requirement but the device would be a pointless addition without some type of logging. It is a requirement if you want to be able to save your weather data and/or upload to other services not natively supported by your PWS. I.e. the device itself does not have built-in data logging capabilities. Logging to a website using the PHP method does not require anything to run locally. Logging locally would require a computer that hosts the logging script to be operational at all times. This can be handled perfectly fine by a Raspberry PI or similar.

Quote
The other point I wanted to be clear on is what, if anything, is impacted by ATMOCOM specifically in regard to the WS-2902A's Display Console as configured through AWNET? Will ATMOCOM interfere in the physical operation of the Ambient system?

It does not interfere with anything. The ATMOCOM device is in essence a special purpose WiFi network router with customized features for sharing data.

Quote
Lastly, and not meaning to sound discouraging and I hate even to bring it up, but will running ATMOCOM conflict with any licensing agreements?

Each upload by your PWS to a cloud service such as AWNET has data records such as wind, temperature etc. in a custom format that is different for each service. At present, only WU data format is recognized by the parsing software so AWNET or any of the other protocols are not used and just passes through to their respective destination. That said, there is nothing really that would relate to licensing other than the protocol format itself which may or may not be proprietary. But we are not using the protocols for any purposes other than intended or modifying the data, we're just parsing out the raw data. This raw data is not owned by the services that you upload to, it is yours. The WU protocol is officially published as are most other protocols so there is no dark web hacking going on.

Offline kbellis

  • Forecaster
  • *****
  • Posts: 361
Re: Data collection for WiFi-only PWS - beta testers wanted!
« Reply #5 on: January 02, 2019, 04:27:34 PM »
Thank you Relko for the thorough reply. I look forward to testing the ATMOCOM device when it arrives - email or PM details at your convenience.

Kind regards,

Kelly

Offline Tilman

  • Member
  • *
  • Posts: 1
Re: Data collection for WiFi-only PWS - beta testers wanted!
« Reply #6 on: May 06, 2019, 04:03:30 AM »
Hello,
is there still a way to get the hardware (e.g. participate in the beta tests)?
I own a W380 Ventus station and I am not really happy with logging to WWW only.
Even, if the provided WSTool App is really b...
Thks.
Tilman

UPDATE: Just ordered the released HW from the Astrogebnic Webschop...
« Last Edit: May 06, 2019, 05:01:16 AM by Tilman »

Offline relko

  • Member
  • *
  • Posts: 30
  • Astrogenic Systems
    • Astrogenic Systems
Re: Data collection for WiFi-only PWS - beta testers wanted!
« Reply #7 on: May 06, 2019, 05:34:40 AM »
Hello,
is there still a way to get the hardware (e.g. participate in the beta tests)?
I own a W380 Ventus station and I am not really happy with logging to WWW only.
Even, if the provided WSTool App is really b...
Thks.
Tilman

Hi,

The product is now available for sale, however, there is some bad news for people living in Germany; Due to a new packaging law (VerpackG) that was put in effect January 1 this year, my company does not sell any physical products to Germany right now. The new law requires mandatory registration with the Central Packaging Registry which includes monthly fees, exactly how much is unclear but in the order of a few hundred euros annually, just to be able to sell a single unit. It doesn't matter if the price of the product is 1 or 100 Euros. If we should fail to register and sell anyway the penalties are upwards of 200000 Euros.

As you can understand, it is unfeasible for most small organizations that sell limited amounts of a niche product to trade in Germany. We're looking into ways of solving the problem but for the time being we are unfortunately unable to sell to any customers living in Germany.

Offline kbellis

  • Forecaster
  • *****
  • Posts: 361
Re: Data collection for WiFi-only PWS - beta testers wanted!
« Reply #8 on: May 07, 2019, 04:27:00 AM »
... there is some bad news for people living in Germany; Due to a new packaging law (VerpackG) that was put in effect January 1 this year, my company does not sell any physical products to Germany right now. The new law requires mandatory registration with the Central Packaging Registry which includes monthly fees, exactly how much is unclear but in the order of a few hundred euros annually, just to be able to sell a single unit. It doesn't matter if the price of the product is 1 or 100 Euros. If we should fail to register and sell anyway the penalties are upwards of 200000 Euros.

That without question sounds unfortunate Relko, but isn't the German law also in direct conflict with the personal, civic, political, economic, and social rights of citizens in the European Union?

Hypothetically, what if Tilman's grandson who lives in Denmark wanted to send his grandfather a gift, is he in trouble? Or what if instead of being his grandson, he was a Danish reseller and he sold an Atmocom to Tilman?

Offline relko

  • Member
  • *
  • Posts: 30
  • Astrogenic Systems
    • Astrogenic Systems
Re: Data collection for WiFi-only PWS - beta testers wanted!
« Reply #9 on: May 07, 2019, 05:28:22 AM »
... there is some bad news for people living in Germany; Due to a new packaging law (VerpackG) that was put in effect January 1 this year, my company does not sell any physical products to Germany right now. The new law requires mandatory registration with the Central Packaging Registry which includes monthly fees, exactly how much is unclear but in the order of a few hundred euros annually, just to be able to sell a single unit. It doesn't matter if the price of the product is 1 or 100 Euros. If we should fail to register and sell anyway the penalties are upwards of 200000 Euros.

That without question sounds unfortunate Relko, but isn't the German law also in direct conflict with the personal, civic, political, economic, and social rights of citizens in the European Union?

Hypothetically, what if Tilman's grandson who lives in Denmark wanted to send his grandfather a gift, is he in trouble? Or what if instead of being his grandson, he was a Danish reseller and he sold an Atmocom to Tilman?

IMO the German law is certainly in conflict with the EU single market idea which guarantees free movement of goods. It doesn't apply to individuals so the grandson would be safe, but not if he is acting as a commercial reseller. Every electronic sample or product sold/given away is predetermined to have some form of packaging and therefore the seller/manufacturer must register for package recycling. No ifs or butts.

At any rate, the immediate problem has been solved. Mr. Tilman is by chance going on a trip to Spain shortly and there will be one Atmocom device waiting at the hotel :)

Offline mcrossley

  • Forecaster
  • *****
  • Posts: 1128
    • Wilmslow Astro
Re: Data collection for WiFi-only PWS - beta testers wanted!
« Reply #10 on: May 07, 2019, 05:47:35 AM »
As I understand it (in a very primitive way) registration is free and mandatory for all commercial sellers, and you only need to be part of a recognised recycling scheme if you exceed certain (quite hefty) tonnages of packing material types each year.
Mark

Offline relko

  • Member
  • *
  • Posts: 30
  • Astrogenic Systems
    • Astrogenic Systems
Re: Data collection for WiFi-only PWS - beta testers wanted!
« Reply #11 on: May 07, 2019, 07:21:16 AM »
As I understand it (in a very primitive way) registration is free and mandatory for all commercial sellers, and you only need to be part of a recognised recycling scheme if you exceed certain (quite hefty) tonnages of packing material types each year.

Not any more. As of 2019 you now also need a license with what they call a certified Dual System. From what I understand this is the organization who is ultimately responsible for picking up and recycling your package and who reports independently to the central registry authority. To obtain this license you have to pay up monthly fees. Then you also need to report twice a year your projected and actual sales to the central registry. Failure to report or incorrect figures = 1,000,000 Euros penalty. Ignoring the law = double that penalty. For my business it is just not worth the hassle.

Offline mcrossley

  • Forecaster
  • *****
  • Posts: 1128
    • Wilmslow Astro
Re: Data collection for WiFi-only PWS - beta testers wanted!
« Reply #12 on: May 07, 2019, 08:45:05 AM »
As I understand it (in a very primitive way) registration is free and mandatory for all commercial sellers, and you only need to be part of a recognised recycling scheme if you exceed certain (quite hefty) tonnages of packing material types each year.

Not any more. As of 2019 you now also need a license with what they call a certified Dual System. From what I understand this is the organization who is ultimately responsible for picking up and recycling your package and who reports independently to the central registry authority. To obtain this license you have to pay up monthly fees. Then you also need to report twice a year your projected and actual sales to the central registry. Failure to report or incorrect figures = 1,000,000 Euros penalty. Ignoring the law = double that penalty. For my business it is just not worth the hassle.
Ah, it is just the requirement to send a Completeness Statement that is exempt for small businesses. What a pain.
Mark

Offline droiddk

  • Forecaster
  • *****
  • Posts: 334
Re: Data collection for WiFi-only PWS - beta testers wanted!
« Reply #13 on: May 07, 2019, 10:54:25 AM »
Cool device!

Like the idea, but I do not upload to WU (and dont want to). Can it get the data and stop it from getting to WU?

Regards


Offline kbellis

  • Forecaster
  • *****
  • Posts: 361
Re: Data collection for WiFi-only PWS - beta testers wanted!
« Reply #14 on: May 07, 2019, 02:42:52 PM »
Cool device!

Like the idea, but I do not upload to WU (and dont want to). Can it get the data and stop it from getting to WU?

Regards

That's a great question!  [tup]

No, presently you can't because the data intended for remote web servers and local databases, are predicated on the WU format; likewise, the weewx driver that Arthur Emerson wrote for ATMOCOM. I wasn't sure about it at first and just finished an experiment that proved this to be true.

Another experiment could be done, though I'm pretty sure I don't want to hassle with WU if things go south, and that would be to go into WU member devices, delete my device on their webpage, but leave the AWNET settings; i.e., KSTATION_ID and key on my mobile devices AWNET for the WS-2902A, or the settings directly entered on the WS-2000 intact. The idea being tested: Can the WU_formatted_stream_packets still stream but then just be ignored by WU?

BTW, I've just today written "Atmocom - A beta tester's review" in case you're interested.

Offline droiddk

  • Forecaster
  • *****
  • Posts: 334
Re: Data collection for WiFi-only PWS - beta testers wanted!
« Reply #15 on: May 07, 2019, 03:21:41 PM »


Another experiment could be done, though I'm pretty sure I don't want to hassle with WU if things go south, and that would be to go into WU member devices, delete my device on their webpage, but leave the AWNET settings; i.e., KSTATION_ID and key on my mobile devices AWNET for the WS-2902A, or the settings directly entered on the WS-2000 intact. The idea being tested: Can the WU_formatted_stream_packets still stream but then just be ignored by WU?


And that is also a good question  :grin: :grin:

Regards

Offline relko

  • Member
  • *
  • Posts: 30
  • Astrogenic Systems
    • Astrogenic Systems
Re: Data collection for WiFi-only PWS - beta testers wanted!
« Reply #16 on: May 07, 2019, 04:32:27 PM »
You can most likely fake WU uploads by providing a non-existing account in the PWS settings. I did some tests a while ago but will do some more just to confirm.

Offline droiddk

  • Forecaster
  • *****
  • Posts: 334
Re: Data collection for WiFi-only PWS - beta testers wanted!
« Reply #17 on: May 07, 2019, 04:47:17 PM »
You can most likely fake WU uploads by providing a non-existing account in the PWS settings. I did some tests a while ago but will do some more just to confirm.

Ok.

Could you add a option to block upload to WU?

Regards

Offline kbellis

  • Forecaster
  • *****
  • Posts: 361
Re: Data collection for WiFi-only PWS - beta testers wanted!
« Reply #18 on: May 08, 2019, 07:33:12 AM »
You can most likely fake WU uploads by providing a non-existing account in the PWS settings. I did some tests a while ago but will do some more just to confirm.

Yep, that works! All I did was simply change the last two numbers of my WU KSTATION_nn that's associated with my WS-2902A, leaving the key value unchanged, all of this being accomplished via the AWNET v4.1.1 app on my Android device. The SQLite db didn't skip a single packet logged in /home/pi/atmocom/wxdb/wx201905.db using the standalone binary atmoudp36.bin

In a separate case and a different SQLite db, the intercept driver for weewx is discriminating between the faux WU KSTATION_nn due to the fact that the /etc/weewx/weewx.conf actually spells out (maps) a specific WU KSTATION_NN, faux or otherwise.

Code: [Select]
$ sudo /etc/init.d/weewx stop
Find [KSTATION_NN] & Replace [WU KSTATION_nn] the 11 instances in the weewx.conf, save.

Code: [Select]
$ sudo /etc/init.d/weewx start
After that, we're back in business :grin: - data is again being written to /var/lib/weewx/weewx.sdb

Offline relko

  • Member
  • *
  • Posts: 30
  • Astrogenic Systems
    • Astrogenic Systems
Re: Data collection for WiFi-only PWS - beta testers wanted!
« Reply #19 on: May 09, 2019, 05:09:29 AM »
You can most likely fake WU uploads by providing a non-existing account in the PWS settings. I did some tests a while ago but will do some more just to confirm.

Ok.

Could you add a option to block upload to WU?

Regards

Not entirely. At least the FOSHK derivatives (WS2902, WS2000 etc.) will not send any data until a successful DNS lookup against WU server is made. If it fails the station will think it is offline and not send any data.

There are "hacky" ways of spoofing a DNS lookup result to fool the station into thinking it is online, but this is quite involved and possibly outside the capacity of the onboard microcontroller in terms of memory.

Another option, again in the case of FOSHK derivatives, is to skip uploading to WU and choose Metoffice WOW or WeatherCloud instead. These protocols are not yet implemented in the Atmocom support scripts but will be in time. The update frequency for WOW (every 5.5 minutes) and WC (every 10 minutes) is also much lower than WU (up to every 30 seconds depending on exact model).

@kbellis: Many thanks for confirming this. It makes sense, the WU station ID and password is in the same URL query as the data so an incorrect login wouldn't be detected until after the station has fired off a HTTP GET request which can be picked off by Atmocom.