Author Topic: WiFiLogger - Connect your Davis console directly to the Internet via WiFi  (Read 109219 times)

0 Members and 2 Guests are viewing this topic.

Offline havtrail

  • Senior Contributor
  • ****
  • Posts: 206
    • Haverford Weather Station
Why does it currently work fine on deprecated site? Just curious.
Sorry. I don't have any answer for that. I was testing current WU.
Maybe they used timestamp not from sample from WiFiLogger, but from WU system.

I believe this is true. My system reports very late to WU, and my observations over time have shown me that the WU "Last Updated...Ago" used to be based on the time that WU received the update (old system), and now (new system) is based on the timestamp of the data logging event on my PWS.

Rich K.
Onset HOBO RX2102 Cellular
https://www.havtrail.com/weather/
NEWA https://newa.cornell.edu Haverford, PA

Offline medic29

  • Weather Watcher
  • Member
  • *
  • Posts: 10
Good morning from Central Illinois in the USA.  It is a brisk, sunny day on the prairie.

I have a couple of issues I think arise from the realtime.txt generated from the WFL. Attached you will see a pic of the data received. Attached below is a screenshot of the realtime.txt file which the WFL sent to my web site.
 [ You are not allowed to view attachments ]

The data template for my site which is based off the Cumulus template processes this information in this array.  [ You are not allowed to view attachments ]

The template is looking at field 21 for the data for yesterday's total rain amount and it shows no data.

I then looked at the information on this site for the realtime.txt file and the data it is supposed to contain.
 [ You are not allowed to view attachments ]

Me being new to this.  I assumed it would be the responsibility of the template to take the information it received from the realtime.txt files and create the totals on it's own, but apparently this is something the weather station already performs and she be outputted.

Can this be corrected?  Or is the data somewhere else in the file??

Thanks,
Rick
Rick

Offline WiFiLogger

  • Forecaster
  • *****
  • Posts: 731
The template is looking at field 21 for the data for yesterday's total rain amount and it shows no data.
Hi Rick

Davis console don't contain such value. I have made realtime.txt because there are some templates for it, but it has a fields which console don't have.
My job is to transfer data from console to the Internet in convinient way, but I am not here to calculate, or approximate any weather value.

I can think about your problem, but it won't be easy. WiFiLogger is like a pipe, transfer and don't accumulate any data. Console is to accumulate data.
I could save in RAM yesterday rain, but WiFiLogger has right to reboot when it will be needed, even more than once per day.
So this value could work, but sometimes will be erased. So this is not very good method to solve this little problem.


About template I will start new thread about this, because I would like to produce template website to use with WiFiLogger.

Offline wvdkuil

  • Wim van der kuil
  • Forecaster
  • *****
  • Posts: 1986
    • My PWS at Leuven Belgium Europe
The template is looking at field 21 for the data for yesterday's total rain amount and it shows no data.
Hi Rick

Davis console don't contain such value. I have made realtime.txt because there are some templates for it, but it has a fields which console don't have.
My job is to transfer data from console to the Internet in convinient way, but I am not here to calculate, or approximate any weather value.

I can think about your problem, but it won't be easy. WiFiLogger is like a pipe, transfer and don't accumulate any data. Console is to accumulate data.
I could save in RAM yesterday rain, but WiFiLogger has right to reboot when it will be needed, even more than once per day.
So this value could work, but sometimes will be erased. So this is not very good method to solve this little problem.


About template I will start new thread about this, because I would like to produce template website to use with WiFiLogger.

Other weather-programs such as Weastherlink.exe or Meteohub also has no yesterday data. Small and large templates can cope with that already for years.

There are  already two "look-a-like"  templates who can process realtime.txt.  So the WiFiLogger is already supported by both.
I will adapt my HWS-wd34 template to cope with that one missing item. No big deal.

It was a ""Tower of Babel" confusion as @medic29 used Cumulus as his weather-program.
If I had known before that he used WiFiLogger, I would have checked the supported fields in realtime.txt, as i did with Weewx  and WeatherCat.

Open questions:
The console normally delivers a "pressure-trend" or am I wrong?  I see only the ---
When there is no UV/Solar sensor those values are replaced with ---  or with 000  ?

Wim
« Last Edit: June 13, 2019, 03:10:36 PM by wvdkuil »

Offline medic29

  • Weather Watcher
  • Member
  • *
  • Posts: 10

It was a ""Tower of Babel" confusion as @medic29 used Cumulus as his weather-program.
If I had known before that he used WiFiLogger, I would have checked the supported fields in realtime.txt, as i did with Weewx  and WeatherCat.

Open questions:
The console normally delivers a "pressure-trend" or am I wrong?  I see only the ---
When there is no UV/Solar sensor those values are replaced with ---  or with 000  ?

Wim

Sorry Wim,

I didn't see an area to show I was getting the data from the WFL.  I assumed (incorrectly it sounds) that the realtime.txt file generated by the WFL was the same as what was generated by Cumulus program.

My apologies.

Rick
Rick

Offline medic29

  • Weather Watcher
  • Member
  • *
  • Posts: 10

Davis console don't contain such value. I have made realtime.txt because there are some templates for it, but it has a fields which console don't have.
My job is to transfer data from console to the Internet in convinient way, but I am not here to calculate, or approximate any weather value.

I can think about your problem, but it won't be easy. WiFiLogger is like a pipe, transfer and don't accumulate any data. Console is to accumulate data.
I could save in RAM yesterday rain, but WiFiLogger has right to reboot when it will be needed, even more than once per day.
So this value could work, but sometimes will be erased. So this is not very good method to solve this little problem.


About template I will start new thread about this, because I would like to produce template website to use with WiFiLogger.

Wojciech, so the Davis console doesn't tally the rainfall amount per 24 hours?  It displays the amount on the screen; rather it displays the "storm" amount.  Also the monthly and annual amounts.  Is any of this data in the wlfex.json file??  Would that make it easier to obtain the data??  Just throwing stuff out there, not that I have any clue.  LOL

I appreciate all of the work you have and continue to put into the WFL.  It has been an incredible addition to my weather station.
Rick

Offline johnd

  • Forecaster
  • *****
  • Posts: 4827
    • www.weatherstations.co.uk
Wojciech, so the Davis console doesn't tally the rainfall amount per 24 hours?  It displays the amount on the screen; rather it displays the "storm" amount.  Also the monthly and annual amounts.  Is any of this data in the wlfex.json file??  Would that make it easier to obtain the data??  Just throwing stuff out there, not that I have any clue.

Is there some confusion creeping in here? The console has values for daily (midnight-to-midnight as the new daily total accumulates, but not a moving retrospective 24-hour total), monthly and annual rain and these values should be in the WFL realtime.txt (eg yearly rain in element 20 of the array (zero-based) but field 21 of the reaLtime listing (1-based)). It should also have a value for Bar trend.

The console does not have (easily accessible) values for some other parameters like yesterday's total, running 24hr rain (as distinct from storm rain, which has a separate definition) etc.
« Last Edit: June 14, 2019, 09:09:27 AM by johnd »
Prodata Weather Systems
Prodata's FAQ/support site for Davis stations
Includes many details on 6313 Weatherlink console.
UK Davis Premier Dealer - All Davis stations, accessories and spares
Cambridge UK

Sorry, but I don't usually have time to help with individual issues by email unless you are a Prodata customer. Please post your issue in the relevant forum section here & I will comment there if I have anything useful to add.

Offline WiFiLogger

  • Forecaster
  • *****
  • Posts: 731
Wojciech, so the Davis console doesn't tally the rainfall amount per 24 hours?  It displays the amount on the screen; rather it displays the "storm" amount.  Also the monthly and annual amounts.  Is any of this data in the wlfex.json file??  Would that make it easier to obtain the data??  Just throwing stuff out there, not that I have any clue.  LOL

I appreciate all of the work you have and continue to put into the WFL.  It has been an incredible addition to my weather station.

There is Daily rain, you can set clear moment: 0:00 or 9:00 am
24 hour rain
1 hour rain
Month rain
Year total.

All have been shown on WiFiLogger page like this: https://wifilogger.net/station/php/
This is small html page to show your wflexp.json
https://wifilogger.net/station/php/wflexp.json

realtime.txt has to be compatible with Cumulus data, because it is theirs own export file. Cumulus is on market for many years, that why there is many ready to go HTML pages to display weather. So I made file like this, but when there was no such data in Davis console, then I omit this value. So maybe there is about 30% lack of data, but this is better, than to put there not accurate data.
After that I have made wflexp.json to export all avaible data from console. This file contains real time data + hilows + some settings and logger and console info.
JSON because this is common internet transfer method with many libraries to use, PHP, JavaScript, .NET, Java etc.

I have also make mistake in this file and I don't know if I should repair that now.
I have used "---" because this is regular DATA NOT AVAIBLE indicator in Davis software.
JSON has "null" for that, no big difference, but libraries wants "null", not "---"

Offline wvdkuil

  • Wim van der kuil
  • Forecaster
  • *****
  • Posts: 1986
    • My PWS at Leuven Belgium Europe
The HWS template is adapted for WiFilogger realtime.txt and in test at the beta test site
Tomorrow I generate a new beta-test version to be downloaded for beta-testing by users themselves.

The forthcoming normal release (July-release) will include Wifilogger support also

I hope I am allowed to use the wifilogger-logo as that saves me the hassle of making one myself

The next October release will include the json file, just need to check which fields to use.

Wim

P.S. The realtime data comes from a wifilogger owners site and is downloaded regularly
Downloading the data for the graphs will cost to much bandwidth,  I use my own  WU data for now.
« Last Edit: June 17, 2019, 11:02:39 AM by wvdkuil »

Offline WiFiLogger

  • Forecaster
  • *****
  • Posts: 731
I hope I am allowed to use the wifilogger-logo as that saves me the hassle of making one myself
No problem with that. There is no copyrights on it.

I am thinking and thinking about custom MySQL scirpt and template for WiFiLogger. Maybe backend PHP script + your template is an answer.
I could order PHP script to accumulate data from WiFiLogger (Davis console) into MySQL.
- Real time data - 3 seconds (like rapidfire)
- HiLows - every 3-5 minutes with accumulating last day entry. 365 hilows per year for very nice statistic data.
- Archive records with charts. This besides on user. From 1 minute interval, up to 2 hour, but default is 30 minutes. In my opinion default should be 15 minutes.

From this MySQL you can take what you want. I have to know what is your needs, before order this PHP software.
My goal is to display for user all Davis data, including extra temp, hum, leaf, soil etc.
For regular data like outside temp, out hum, wind etc, there is a planty of services, one better from other.

Offline gvspfc

  • Member
  • *
  • Posts: 24
    • Blumenau/Brazil Wunderground
I hope I am allowed to use the wifilogger-logo as that saves me the hassle of making one myself
No problem with that. There is no copyrights on it.

I am thinking and thinking about custom MySQL scirpt and template for WiFiLogger. Maybe backend PHP script + your template is an answer.
I could order PHP script to accumulate data from WiFiLogger (Davis console) into MySQL.
- Real time data - 3 seconds (like rapidfire)
- HiLows - every 3-5 minutes with accumulating last day entry. 365 hilows per year for very nice statistic data.
- Archive records with charts. This besides on user. From 1 minute interval, up to 2 hour, but default is 30 minutes. In my opinion default should be 15 minutes.

From this MySQL you can take what you want. I have to know what is your needs, before order this PHP software.
My goal is to display for user all Davis data, including extra temp, hum, leaf, soil etc.
For regular data like outside temp, out hum, wind etc, there is a planty of services, one better from other.

Love this idea!

Offline R_o_B

  • WxElement panel
  • Senior Member
  • *****
  • Posts: 85
Wojciech, do you have something already planned (a PHP script) that would save the data directly from the WiFiLogger to a SQL database, somewhat similar to what Weather Display does? Would there be enough memory left on the WiFiLogger chip to incorporate the SQL option inside the WiFiLogger chip?

Wojciech could I ask you for a favour: Could you make available a web page where all the most recent firmware updates for the WiFiLogger would be available as well as the version number and the release date of the last firmware update ... please.  ;)
R_o_B
---
eMail: weather@herray.net

Offline WiFiLogger

  • Forecaster
  • *****
  • Posts: 731
Wojciech, do you have something already planned (a PHP script) that would save the data directly from the WiFiLogger to a SQL database, somewhat similar to what Weather Display does? Would there be enough memory left on the WiFiLogger chip to incorporate the SQL option inside the WiFiLogger chip?
I have made this JSON export file and second one with archive records. This will be transport layer.
I don't know PHP. I could make script on tutorial level to save data to MySQL, but I think I will hire a company to make this.
I wanted to make also template to view this data, but I think backend should be made first.
I have to think about inteface to take out data from MySQL.

What I want from MySQL?
- One owner of MySQL. Multiple stations. - farmer with 5 stations. - Yacht club with 5 stations. - Owner of website who could invite friends.
   In general typical hosting with MySQL and PHP is not very good for data aquisition, but for 40 stations should be ok.
- Real Time Data - only one entry per station(current). RapidFire skill.
- HiLows - updated every 3 minutes. Last entry from a day should be archived. HiLows are very nice statistic data, we should not lose it.
- Archive records from datalogger memory fully saved to MySQL. View with charts.
- Basic setup from station. Units, latitude, longitude, elevation, IP, etc.
It is very similar to WL.COM, because they take all data from console. WiFiLogger was built to export all data from console, there is no surprise here.

There is also GRAPH data in console. I have not used it. Probably there is also very nice data. Maybe it is time to examine it?

Wojciech could I ask you for a favour: Could you make available a web page where all the most recent firmware updates for the WiFiLogger would be available as well as the version number and the release date of the last firmware update ... please.  ;)
I could try, but I can't promise.
I do know that I make some mess in firmware versions, but this allow me to react fast. When I will have to make additional documents, because I have change one line in firmware, this will cause decrease of firmware updates.

I can't published firmware at the moment. This could damage WiFiLoggers with version 1.xx.
Version 2.0 has this HTTP update system, for version 1.xx there is only one firmware published to migrate to 2.0.

Offline R_o_B

  • WxElement panel
  • Senior Member
  • *****
  • Posts: 85
I don't know PHP. I could make script on tutorial level to save data to MySQL, but I think I will hire a company to make this.
Wojciech, that might cost you a lot of money - are you that rich?  :oops:

Between this 'WXforum.net' forum and the 'Weather-Watch.com' forum, I am sure that you could find probably a half-dozen or more of very knowledgeable PHP and SQL programmers that would be pleased to give you some of their time to create and develop the needed/required scripts.

Wojciech could I ask you for a favour: Could you make available a web page where all the most recent firmware updates for the WiFiLogger would be available as well as the version number and the release date of the last firmware update ... please.  ;)
I could try, but I can't promise.
I do know that I make some mess in firmware versions, but this allow me to react fast. When I will have to make additional documents, because I have change one line in firmware, this will cause decrease of firmware updates.
Wojciech, everyone make mistake in their code (such as Microsoft, for example  :roll: ). So what, you quickly need to change the latest release of the firmware because you found an error in the code, you put the new version of the firmware online with a slightly different version number - such as 2.20 for the first version and 2.20.1 for the corrected version and ... 2.20.99 for another corrected version, and so on, as long as there are no major modifications in the code. You also place online and inside the release firmware ZIP file a 'changelog' text file with the proper version number, the proper release date and the most recent modifications.

As for versions 1.xx - well, they are all 'deprecated/obsoleted/unsupported' versions - the only way for someone to get support would be to update/upgrade the WiFiLogger to version 2.xx.  :-|
R_o_B
---
eMail: weather@herray.net

Offline droiddk

  • Forecaster
  • *****
  • Posts: 334
Between this 'WXforum.net' forum and the 'Weather-Watch.com' forum, I am sure that you could find probably a half-dozen or more of very knowledgeable PHP and SQL programmers that would be pleased to give you some of their time to create and develop the needed/required scripts.

Agree. Or simple add the option for the user to input an URL and then let the wifi Logger make a HTTP Post Request (or GET as the Wunderground protocol) with the data to this URL. Then the user can do with the data what he wants from the PHP $_POST array, for example generating a JSON-file for "real time" live-updates using AJAX on a website or inserting the data into MySQL. The new wifi-firmware for the Fine Offset wifi consoles (and clones) have exactly this option, its great! If you want to try this route then contact me. I don't have a functional Pro2 setup atm but I have 2 fully Pro2 Plus systems on the shelf and I can power up a console for testing if you want (don't have the WIFI-logger).

But I don't understand some parts of this:

-  In general typical hosting with MySQL and PHP is not very good for data aquisition, but for 40 stations should be ok.

MySQL not good for data? Not good for small data, why not?

- Real Time Data - only one entry per station(current). RapidFire skill.

One new row every 2.5 second? That is heavy and not a job for MySQL. Instead PHP can generate a JSON-file with real time updates.

- HiLows - updated every 3 minutes. Last entry from a day should be archived. HiLows are very nice statistic data, we should not lose it.

Ok. Sounds like a job for a second table.

Regards

« Last Edit: June 21, 2019, 02:29:35 PM by droiddk »

Offline 92merc

  • BismarckWeather.net
  • Forecaster
  • *****
  • Posts: 1308
  • BismarckWeather.net
    • BismarckWeather.net
When it comes to SQL, If I'm not mistaken, the Cumulus software has one table for realtime data.  Not sure of the update frequency there.  Then a second table for 10 minute archive of the data.

This would make more sense to me since then your PHP would just refresh the live data table.
https://www.BismarckWeather.net
Davis VP2, Cumulus, WeatherDisplay, Blitzortung, Saratoga Scripts, NOAA Stream via PI

Offline droiddk

  • Forecaster
  • *****
  • Posts: 334
When it comes to SQL, If I'm not mistaken, the Cumulus software has one table for realtime data.  Not sure of the update frequency there.  Then a second table for 10 minute archive of the data.

This would make more sense to me since then your PHP would just refresh the live data table.

That could be.  [tup]
But no need for a table (even with just one row that is being updated) just for real time updates if the initial update is through PHP, for this a local JSON-file is just fine.

Offline 92merc

  • BismarckWeather.net
  • Forecaster
  • *****
  • Posts: 1308
  • BismarckWeather.net
    • BismarckWeather.net
Yeah, I don't have my "live" section enabled in Cumulus.  But I do have it enabled for the 10 minute upload to SQL.

If a JSON file is simpler, that's the way to go.
https://www.BismarckWeather.net
Davis VP2, Cumulus, WeatherDisplay, Blitzortung, Saratoga Scripts, NOAA Stream via PI

Offline WiFiLogger

  • Forecaster
  • *****
  • Posts: 731
table Admins - users
table WiFiLoggers - list with connected units, info, settings etc.
table RTD - real time data, only one row for one logger, only current data, no archive here. No problem with MySQL and 3sec. here. Row will be updated, not added.
table HiLows - hilows - current + archive (last entry from each day)
table Archive - here will be stored history, from logger memory. Depends on logger, from 1 min up to 2h interval.

I will split JSON file to generate smaller traffic. Now it contains settings, real time data and hilows. So I will devide it. I will left also this one which is now.

User will have charts from Archive. This data has little one fault, wind direction is in 1-16 scale, not 0-360 degree. It is ok, because it is average, but will not look nice like 0-360.

In Davis system extra temperatures, like soil temp, leaf temp, extra temp, those are in full F scale, no decimal part. In MySql and whole system it will be with decimal part. I think I have little idea how to use this data.
Any other data will be 1:1 with Davis specification.

Offline WiFiLogger

  • Forecaster
  • *****
  • Posts: 731
MySQL not good for data? Not good for small data, why not?

Not MySQL, but hosting services. You have some of them for free, for other you will have to pay $30-$50 per year.
Those are for simple webpage and email. I do believe that up to 40 stations should be ok.
If you would like to have cloud for 1mln users per day, then you will have to pay a lot.

One new row every 2.5 second? That is heavy and not a job for MySQL. Instead PHP can generate a JSON-file with real time updates.
In RTD table there will be only one row for station, only current data will be there, so this row will be updated, not added.
Archive table will be populated with logger data.

I think WU is recording every 5 minutes, even if you are using RapidFire, but they don't operate with archive memory from logger.
WL.COM has charts from archive. I think WL.COM don't record real time data.

Offline droiddk

  • Forecaster
  • *****
  • Posts: 334
table Admins - users
table WiFiLoggers - list with connected units, info, settings etc.
table RTD - real time data, only one row for one logger, only current data, no archive here. No problem with MySQL and 3sec. here. Row will be updated, not added.
table HiLows - hilows - current + archive (last entry from each day)
table Archive - here will be stored history, from logger memory. Depends on logger, from 1 min up to 2h interval.

I will split JSON file to generate smaller traffic. Now it contains settings, real time data and hilows. So I will devide it. I will left also this one which is now.

User will have charts from Archive. This data has little one fault, wind direction is in 1-16 scale, not 0-360 degree. It is ok, because it is average, but will not look nice like 0-360.

In Davis system extra temperatures, like soil temp, leaf temp, extra temp, those are in full F scale, no decimal part. In MySql and whole system it will be with decimal part. I think I have little idea how to use this data.
Any other data will be 1:1 with Davis specification.

Ok. Good luck!

Offline WiFiLogger

  • Forecaster
  • *****
  • Posts: 731
I have made first step in PHP&MySQL job.
"Logger" table is made
PHP part for "logger" is partially done.

I am preparing JSON export file in WiFiLogger and MySQL table in same time.

Will be here someone who knows databases and could help?

Firts thing to ask:
I have made a few columns for time purpose
"davis_timestamp" - same number which is in archive records in console. This is local time in Davis format.
"time" - local time from davis time stamp in MySQL format
"date" - local date from davis time stamp in MySQL format
"time_utc" - recalculated from local time to UTC in Unix epoch number

Davis time stamp is important for WiFiLogger.
UTC time is universal
Should I erase "time" and "date"? Or merge it to one DateTime cell?

Second thing. Cumulus, WeatherLink are downloading data from WeatherLink.com in binary davis format.
Should I add column with this raw binary data?

I have also made BIGINT column as WiFiloggerID. It is MAC address (48bits). It is very convenient, because it is unique id comes from WiFiLogger and can be used in any table, but it is BIGINT not INT.
Will it be ok to use this BIGINT as ID in database?

Offline wvdkuil

  • Wim van der kuil
  • Forecaster
  • *****
  • Posts: 1986
    • My PWS at Leuven Belgium Europe
. . . shortened . . .
Firts thing to ask:
I have made a few columns for time purpose
"davis_timestamp" - same number which is in archive records in console. This is local time in Davis format.
"time" - local time from davis time stamp in MySQL format
"date" - local date from davis time stamp in MySQL format
"time_utc" - recalculated from local time to UTC in Unix epoch number

Davis time stamp is important for WiFiLogger.
UTC time is universal
Should I erase "time" and "date"? Or merge it to one DateTime cell?
You should leave them as is
Quote
Second thing. Cumulus, WeatherLink are downloading data from WeatherLink.com in binary davis format.
Should I add column with this raw binary data?
What can we use that data for ?
Quote
I have also made BIGINT column as WiFiloggerID. It is MAC address (48bits). It is very convenient, because it is unique id comes from WiFiLogger and can be used in any table, but it is BIGINT not INT.
Will it be ok to use this BIGINT as ID in database?
Yes, scripts can cope with that and it would be easier to select one logger if there are multiple.

Succes,
Wim 

Offline WiFiLogger

  • Forecaster
  • *****
  • Posts: 731
Second thing. Cumulus, WeatherLink are downloading data from WeatherLink.com in binary davis format.
Should I add column with this raw binary data?
Quote
What can we use that data for ?

Wim, I am appreciate for professional help.

Raw data can be used to make compatible interface with Cumulus, same which is using to download archive data from WeatherLink.com.
Now Cumulus can download archive data from WeatherLink.com, but if there will be a chance to change host for this operation, then my PHP&MySQL could server same data in same format.

WeatherLink.com is using DID(MAC address) and KEY to give access unit to store data in database.
My PHP&MySQL will use "allow" table with WiFiLoggerID(MAC address). This script is not for cloud service, but for a few WiFiLoggers, so units can be added manually.
If "allow" table will be empty, then all WiFiLogger will be allow to save data in this database.
If "allow" table will be populated with at least one MAC address, then only this one will be saved, other will be rejected.

Offline WiFiLogger

  • Forecaster
  • *****
  • Posts: 731
PHP&MySQL report

Data logger archive memory export to MySQL is almost working.
Unique composite key is on columns: wfl_id and davis_timestamp.
wfl_id is MAC address of WiFiLogger, so it is unique ID of any WiFiLogger.
davis_timestamp is raw date time value taken directly from data logger memory.
Those has to be unique, so user will be no allow to add second time same data. This will trigger warning from database.
Same data can be sent second time to MySQL, but it just won't be added.

Procedure to add logger data has two steps.
1. Ask for last time date from database.
2. Send batch of logger data.

Works ok. I just have to make some minor changes.