Author Topic: Meteobridge PRO often 5 min. of min. data missing in export from RAM-DB **solved  (Read 1579 times)

0 Members and 1 Guest are viewing this topic.

Offline Gyvate

  • Forecaster
  • *****
  • Posts: 3329
I observed a phenomenon I cannot explain - maybe someone here has an idea ...
I'm exporting every hour the data captured by MB Pro from my GW1000 on a minute base.
In the files there is most of the time(the export window is one hour) one, sometimes two, gaps of mostly five (sometimes eight) minutes with no data. Two examples below:
--------------
2020-05-27,00:06,14.5,61,7.1,1032.0,0.0,0.0,101,0.0, 0.00, , , 17.8, 51, 41, 51, 48
2020-05-27,00:07,,,,1032.0,,,,0.0, 0.00, , , 17.8, 51, , 51, 48
2020-05-27,00:08,,,,,,,,, , , , , , , ,
2020-05-27,00:09,,,,,,,,, , , , , , , ,
2020-05-27,00:10,,,,,,,,, , , , , , , ,
2020-05-27,00:11,,,,,,,,, , , , , , , ,
2020-05-27,00:12,,,,,,,,, , , , , , , ,
2020-05-27,00:13,14.5,62,7.2,1031.9,0.0,0.5,101,0.0, 0.00, , , 17.8, 50, 41, 51, 48
--------------------------
2020-05-27,11:29,20.8,41,7.1,1031.6,0.3,0.5,150,0.0, 813.00, , , 21.1, 41, 42, 52, 46
2020-05-27,11:30,20.8,41,7.1,,0.3,0.5,150,, 813.00, , , , , , ,
2020-05-27,11:31,,,,,,,,, , , , , , , ,
2020-05-27,11:32,,,,,,,,, , , , , , , ,
2020-05-27,11:33,,,,,,,,, , , , , , , ,
2020-05-27,11:34,,,,,,,,, , , , , , , ,
2020-05-27,11:35,,,,,,,,, , , , , , , ,
2020-05-27,11:36,,,,,,,,, , , , , , , ,
2020-05-27,11:37,,,,,,,,, , , , , , , ,
2020-05-27,11:38,,,,,,,,, , , , , , , ,
2020-05-27,11:39,21.2,41,7.2,1031.3,0.0,0.0,85,0.0, 605.67, , , 21.0, 41, 42, 52, 46
2020-05-27,11:40,21.6,40,7.3,1031.5,0.4,3.6,99,0.0, 846.51, , , 21.3, 42, 41, 53, 45
--------------------------
data definition of the export script:
$# date, time, temperature[C], humidity[%], dew point[C], sealevel pressure[hPa], avg wind speed[m/s], gust speed[m/s], winddir, rainfall[mm], Solar [W/m2], PM25(1) [ug/m3], PM25(2), extra-temp1 [°C], extra-hum [%], soil moisture (1) [%], SM2, SM3
[YYYY]-[MM]-[DD],[hh]:[mm],[th0temp-avg.1:],[th0hum-avg.0:],[th0dew-avg.1:],[thb0seapress-avg.1:],[wind0avgwind-avg.1:],[wind0wind-max.1:],[wind0dir-avg.0:],[rain0total-sum.1:], [sol0rad-avg.2:], [air0pm-avg.0:], [air1pm-avg.1:], [th1temp-avg.1:], [th1hum-avg.0:], [th20hum-avg.0:], [th21hum-avg.0:], [th22hum-avg.0:]
--------------------------
Sometimes I observe sort of a degradation in data quality before the gap, i.e. there are no more values for all sensors but only for some.

[the two gaps inside the line , , , before the last 4 values are a systematic error in my opinion - they are the (missing) air quality sensor minute values. I have an open post in the Meteobridge PRO section of the Meteohub forum with about 200 views but no reply regarding this issue. (MB PRO - export and graphic depiction of air quality sensors (air0pm, air1pm) fails).But that's a different thing.]
There is no visible periodicity for the occurrence - sometimes 30 minutes, sometimes more.

Any idea where this comes from ?

Looks as if the logger chokes and initiates a restart. In the systems log visible in the system tab there are no respective messages. (Maybe in some other log file ?)

Rebooting the device doesn't bring any change.
Thanks
« Last Edit: September 10, 2020, 02:50:06 PM by Gyvate »
WS2350 1.6.7, GW1000(3) 1.7.7,WH2650 WiFi (2) 1.7.7 (test/backup), GW1100 2.3.1, GW2000(3) 3.1.1, HP2551 1.9.5,5.1.5;HP3500 1.7.2,WS3800 1.2.8, WN1910 1.2.3,WN1980 1.2.3;
Ecowitt WS90(2)1.3.5/1.4.0, WS80(2)1.2.5, WS68, WS69, WH40, WH31, WH31-EP, WN30, WN34L, WN35, WH32, WH32-EP, WH32B, WH57 [Lightning], WH41 [PM2.5], WH51, WH45, WH55
MeteobridgePro(2)[test,prod] 5.8 Mar 01 2024, 15185 - Blake-Larsen Sun Recorder - RPi4/weewx 4.8.0/4.10.2/CumulusMX 3283/Meteobridge RPi4B-2GB(3169)
Barani Meteoshield Pro, MetSpec Rad02 - Ecowitt 5763,34418;WU ISAARB3(WH4000SE),ISAARB22(HP2553), http://meshka.eu

Offline ConligWX

  • Forecaster
  • *****
  • Posts: 843
  • #conligwx
    • conligwx.org
might be worth informing Boris you're having these type of issue.

https://forum.meteohub.de/  or better still email: info(at)smartbedded.com
Regards Simon
Davis Vantage Pro2 Plus (6162UK) • Daytime FARS • WeatherLink Live • AirLink • PurpleAir PA-II-SD • CumulusMX •


Offline Mattk

  • Forecaster
  • *****
  • Posts: 2161
Gaps like that generally indicate there is no data

Offline Gyvate

  • Forecaster
  • *****
  • Posts: 3329
Sure - no data.  8-)
But where ? Not written to the RAM-DB, because the logger didn't do its job properly ? Or because no data received from the GW1000 ? It would of course be helpful, if the logger wrote log entries for such events. But it doesn't, at least not to the system log shown in the web interface. Maybe into some other log file accessible on the OS level of the MB Pro device ...
Then we could identify or narrow down the culprit (MB or GW1000 or the router).

But I have a suspicion. Around the time when these gaps occurred first, I had a firmware update of the GW1000 from 1.5.8 to 1.5.9.
I have a backup device which still runs on 1.5.6.
I have connected this now to the MB Pro and will observe/check the hourly exports for gaps.
If the firmware of the GW1000 is the culprit, the gaps should disappear.
« Last Edit: May 28, 2020, 07:22:03 AM by Gyvate »
WS2350 1.6.7, GW1000(3) 1.7.7,WH2650 WiFi (2) 1.7.7 (test/backup), GW1100 2.3.1, GW2000(3) 3.1.1, HP2551 1.9.5,5.1.5;HP3500 1.7.2,WS3800 1.2.8, WN1910 1.2.3,WN1980 1.2.3;
Ecowitt WS90(2)1.3.5/1.4.0, WS80(2)1.2.5, WS68, WS69, WH40, WH31, WH31-EP, WN30, WN34L, WN35, WH32, WH32-EP, WH32B, WH57 [Lightning], WH41 [PM2.5], WH51, WH45, WH55
MeteobridgePro(2)[test,prod] 5.8 Mar 01 2024, 15185 - Blake-Larsen Sun Recorder - RPi4/weewx 4.8.0/4.10.2/CumulusMX 3283/Meteobridge RPi4B-2GB(3169)
Barani Meteoshield Pro, MetSpec Rad02 - Ecowitt 5763,34418;WU ISAARB3(WH4000SE),ISAARB22(HP2553), http://meshka.eu

Offline weather34

  • Forecaster
  • *****
  • Posts: 1068
    • https://weather34.com/homeweatherstation
please take a look at your export script

avg1, avg2, avg3, ..., avg60: selects average value from the last one to 60 minutes note there are NO decimals given in the example listed on the wiki.

when defining the average note no decimal after the avg i.e you have in some avg.1 if so are you trying to define the average for the last minute ?  try avg1  example [wind0avgwind-avg1:] and so on

whenever ive used the average ive simply used avg1 , avg10, etc never used avg.1

might help might not but try it and see how that goes .brian

Offline ConligWX

  • Forecaster
  • *****
  • Posts: 843
  • #conligwx
    • conligwx.org
please take a look at your export script

avg1, avg2, avg3, ..., avg60: selects average value from the last one to 60 minutes note there are NO decimals given in the example listed on the wiki.

when defining the average note no decimal after the avg i.e you have in some avg.1 if so are you trying to define the average for the last minute ?  try avg1  example [wind0avgwind-avg1:] and so on

whenever ive used the average ive simply used avg1 , avg10, etc never used avg.1

might help might not but try it and see how that goes .brian

Brian.

I think the avg.1 avg.0 etc is the amount of decimal places in the processed value afaik.

its used in chart definitions  https://www.meteobridge.com/wiki/index.php/PRO_Graphs

dataX defines data set "X". "X" should start at 1 and needs to be incremented for each new data set to be used in the chart. Data sets are sensor names as used in template expansions like "th0temp-avg.1" which represents outdoor temperature (th0 sensor) averaged on the chosen resolution and with one decimal precision. Examples: "rain0total-daysum=in.2" represents the cumulated rain fall (rain0 sensor) per day in inch with 2 decimals precision. "rain0total-allsum.1" represents the cumulated rainfall in mm over the total selected time period in 1 decimal precision.
« Last Edit: May 28, 2020, 09:04:32 AM by ConligWX »
Regards Simon
Davis Vantage Pro2 Plus (6162UK) • Daytime FARS • WeatherLink Live • AirLink • PurpleAir PA-II-SD • CumulusMX •


Offline weather34

  • Forecaster
  • *****
  • Posts: 1068
    • https://weather34.com/homeweatherstation
that avg.1 dont look right is the period 1 minute , 10 minute ? defining the average of last 10 minutes for example defined by avg10

if I read the export script op posted humidity for example [th1hum-avg.0:] to me thats looking for humidity last .0 im not seeing a period defined
[th1hum-avg1] would seem appropriate

the wiki note the period is defined directly after i.e val10,max10,min10,avg10,sum10 not seeing a decimal place prior to period value.


Apart from selectors that use absolute, predefined time slots there are also selectors that look for a certain amount of time into the past.
val1, val1, val3, ..., val60: selects the value the sensor has shown one to 60 minutes ago (This is only available for sensors with ID 0 and 1, like "th0temp" or "th1temp", unless you have a Meteobridge PRO).
max1, max2, max3, ..., max60: selects the maximum value from the last one to 60 minutes

min1, min2, min3, ..., min60: selects the minimum value from the last one to 60 minutes
avg1, avg2, avg3, ..., avg60: selects average value from the last one to 60 minutes

For sensors delivering cumulated values like "rain0total" and "sol0evo" the following selectors are defined:
sum1, sum2, sum3, ..., sum60, sum1h, sum2h, sum3h, ..., sum24h: selects summarized delta values from the last one to 60 minutes or one to 24 hours. This is useful to get amount of total rain in a certain time frame: "rain0total-sum60" is rainfall in mm of last 60 minutes.


delta1, delta2, delta3, ..., delta60, delta1h, delta2h, delta3h, ..., delta24h: selects difference between current value and value from one to 60 minutes or one to 24 hours ago. A positive number shows that value has increased, a negative number indicates the value has decreased. This is useful to do trend analysis over various time spans.

think define the decimal after the period syntax


Offline ConligWX

  • Forecaster
  • *****
  • Posts: 843
  • #conligwx
    • conligwx.org
only Boris would be able to confirm what his software can and cannot do with this decimal place though he does use "th0temp-avg.1" in an example. we are purely speculating, though I understand your concern.
Regards Simon
Davis Vantage Pro2 Plus (6162UK) • Daytime FARS • WeatherLink Live • AirLink • PurpleAir PA-II-SD • CumulusMX •


Offline galfert

  • Global Moderator
  • Forecaster
  • *****
  • Posts: 6822
As Weather34 has pointed out, your syntax is wrong.
Surprised you are getting any data at all. You must have more than one service defined. It is the other service that is set for 5 minutes that is actually correct and producing data.
Ecowitt GW1000 | Meteobridge on Raspberry Pi
WU: KFLWINTE111  |  PWSweather: KFLWINTE111
CWOP: FW3708  |  AWEKAS: 14814
Windy: pws-f075acbe
Weather Underground Issue Tracking
Tele-Pole

Offline galfert

  • Global Moderator
  • Forecaster
  • *****
  • Posts: 6822
By the way there is no need to define output for one decimal place as that is the default when formating is not specified. If you need 2 decimal places then you do use the converter...but the way you are going about it is incorrect.

For example here is the correct way to define decimal places
[thb0seapress-act=inHg.2]
Ecowitt GW1000 | Meteobridge on Raspberry Pi
WU: KFLWINTE111  |  PWSweather: KFLWINTE111
CWOP: FW3708  |  AWEKAS: 14814
Windy: pws-f075acbe
Weather Underground Issue Tracking
Tele-Pole

Offline Gyvate

  • Forecaster
  • *****
  • Posts: 3329
Status update:
1. I've exchanged the GW1000 FW 1.5.9 with my backup device FW 1.5.6 and for the past five hours no gaps occurred. Will continue for 48 hours to have a more solid data base.

2. I've agreed with a fellow forum member with a similar hardware setup and running GW1000 FW 1.5.9 to do the same hourly minute-based all sensor exports for 24-48 hours to have a comparison.

3. regarding syntax: Boris stated that for the minute values, the data definition should be e.g. [air0pm-act.1], however this doesn't yield any result, whereas [air0pm-avg.1] does.
He is checking this now in my MB Pro which has been upgraded to Meteobridge 4.3 (May 28 2020, build 13497), FW 1.4 to solve another issue I had with the air quality sensors.
https://forum.meteohub.de/viewtopic.php?f=61&t=14846
The principal issue (air quality sensors can neither be shown in graphs nor be exported) seems to be solved with the update, but Boris is looking into the -act/-avg "thing".
« Last Edit: May 28, 2020, 01:04:16 PM by Gyvate »
WS2350 1.6.7, GW1000(3) 1.7.7,WH2650 WiFi (2) 1.7.7 (test/backup), GW1100 2.3.1, GW2000(3) 3.1.1, HP2551 1.9.5,5.1.5;HP3500 1.7.2,WS3800 1.2.8, WN1910 1.2.3,WN1980 1.2.3;
Ecowitt WS90(2)1.3.5/1.4.0, WS80(2)1.2.5, WS68, WS69, WH40, WH31, WH31-EP, WN30, WN34L, WN35, WH32, WH32-EP, WH32B, WH57 [Lightning], WH41 [PM2.5], WH51, WH45, WH55
MeteobridgePro(2)[test,prod] 5.8 Mar 01 2024, 15185 - Blake-Larsen Sun Recorder - RPi4/weewx 4.8.0/4.10.2/CumulusMX 3283/Meteobridge RPi4B-2GB(3169)
Barani Meteoshield Pro, MetSpec Rad02 - Ecowitt 5763,34418;WU ISAARB3(WH4000SE),ISAARB22(HP2553), http://meshka.eu

Offline Gyvate

  • Forecaster
  • *****
  • Posts: 3329
As Weather34 has pointed out, your syntax is wrong.
Surprised you are getting any data at all. You must have more than one service defined. It is the other service that is set for 5 minutes that is actually correct and producing data.
wondering what is really wrong there - especially as Boris who looked into the data definition didn't mention any faulty part.
To make it clear, could you then write the correct syntax down please - it's not so long a data definition ... ;)

And regarding other services set for 5 minutes .... - which one should that be ? see attachment.
There are no 2 exports running at the same time during the day, only once an hour.
« Last Edit: May 28, 2020, 01:21:34 PM by Gyvate »
WS2350 1.6.7, GW1000(3) 1.7.7,WH2650 WiFi (2) 1.7.7 (test/backup), GW1100 2.3.1, GW2000(3) 3.1.1, HP2551 1.9.5,5.1.5;HP3500 1.7.2,WS3800 1.2.8, WN1910 1.2.3,WN1980 1.2.3;
Ecowitt WS90(2)1.3.5/1.4.0, WS80(2)1.2.5, WS68, WS69, WH40, WH31, WH31-EP, WN30, WN34L, WN35, WH32, WH32-EP, WH32B, WH57 [Lightning], WH41 [PM2.5], WH51, WH45, WH55
MeteobridgePro(2)[test,prod] 5.8 Mar 01 2024, 15185 - Blake-Larsen Sun Recorder - RPi4/weewx 4.8.0/4.10.2/CumulusMX 3283/Meteobridge RPi4B-2GB(3169)
Barani Meteoshield Pro, MetSpec Rad02 - Ecowitt 5763,34418;WU ISAARB3(WH4000SE),ISAARB22(HP2553), http://meshka.eu

Offline galfert

  • Global Moderator
  • Forecaster
  • *****
  • Posts: 6822
Okay the issue isn't that there is another script running it seems.

I think the issue has to do with the lack of availability of data because of the syntax that you are using.

When you say th0temp-avg.1: what you have defined there is temperature average of the last tenth of a minute. The -avg modifier requires a digit to define the time frame to pull data from to get an average. Typical usage for -avg is therefore to use -avg1 or avg2 or avg5 or with whatever minute time frame you are looking to compute an average for. When you use -avg.1 then you are saying 1/10 of minute or 6 seconds. Using that you get no data because there is not enough data to compute an average.

The reason you are having more difficult time with the PM2.5 is because you probably only have the outdoor version which only updates about every 10 minutes.

If you want to upload a data point for every minute then you should use either -avg1 or -act. But you may want to do something else for wind gusts as the NWS defines wind gusts as the maximum wind for the past 10 minutes. But people do their own thing, so I suppose you can define wind gusts as you please although some may object to your definition.

Proper syntax for temperature average past 1 minute in °C with tenths decimal place:
th0temp-avg1

No need to specify that you want the data to show tenths of a degree as that is the default format. You only need to modify the data for decimal place changes if you need something other than showing tenths, like you want whole numbers or you want to show hundredths.
« Last Edit: May 28, 2020, 02:03:34 PM by galfert »
Ecowitt GW1000 | Meteobridge on Raspberry Pi
WU: KFLWINTE111  |  PWSweather: KFLWINTE111
CWOP: FW3708  |  AWEKAS: 14814
Windy: pws-f075acbe
Weather Underground Issue Tracking
Tele-Pole

Offline galfert

  • Global Moderator
  • Forecaster
  • *****
  • Posts: 6822
In case you want temperature average for 1 minute °C in hundredths (2 decimal places)
th0temp-avg1=C.2

Realize though that the following 4 provide equal results:
th0temp-avg1=C.1
th0temp-avg1=.1
th0temp-avg1=C
th0temp-avg1

Because metric is default and showing tenths is default.
« Last Edit: May 28, 2020, 02:53:12 PM by galfert »
Ecowitt GW1000 | Meteobridge on Raspberry Pi
WU: KFLWINTE111  |  PWSweather: KFLWINTE111
CWOP: FW3708  |  AWEKAS: 14814
Windy: pws-f075acbe
Weather Underground Issue Tracking
Tele-Pole

Offline Gyvate

  • Forecaster
  • *****
  • Posts: 3329
Okay the issue isn't that there is another script running it seems.

I think the issue has to do with the lack of availability of data because of the syntax that you are using.

When you say th0temp-avg.1: what you have defined there is temperature average of the last tenth of a minute. The -avg modifier requires a digit to define the time frame to pull data from to get an average. Typical usage for -avg is therefore to use -avg1 or avg2 or avg5 or with whatever minute time frame you are looking to compute an average for. When you use -avg.1 then you are saying 1/10 of minute or 6 seconds. Using that you get no data because there is not enough data to compute an average.

The reason you are having more difficult time with the PM2.5 is because you probably only have the outdoor version which only updates about every 10 minutes.

If you want to upload a data point for every minute then you should use either -avg1 or -act. But you may want to do something else for wind gusts as the NWS defines wind gusts as the maximum wind for the past 10 minutes. But people do their own thing, so I suppose you can define wind gusts as you please although some may object to your definition.

Proper syntax for temperature average past 1 minute in °C with tenths decimal place:
th0temp-avg1

No need to specify that you want the data to show tenths of a degree as that is the default format. You only need to modify the data for decimal place changes if you need something other than showing tenths, like you want whole numbers or you want to show hundredths.
OK - after using the GW1000 with the older firmware no more gaps occurred in the past 6 hours.
After Boris' software update the PM2.5 values also show (which was not the topic of this thread but nice anyway - the update is specifically meant for PM2.5 values).

Now, regarding the syntax the way I read your and Brian's post, there seem to be two different opinions what the syntax means:

avg.1 in your (galfert's) opinion means that the average of 1/10th of a minute is asked for
and the resolution parameter (minute, hour, day) defines what this 1/10th is taken from

whereas the other opinion (and this is how I understood the notation) refers to the formatting and accuracy of the values.
here avg.1 means the average (on whichever basis it is created, probably depends on the resolution selected, but that's an assumption) with one digit after the decimal point.
see wiki graphs  - here th0hum-avg.0 means that 0 = no decimal places after the decimal point

I have run an export with the following definitions for air0pm and air1pm: [air0pm-avg.1:], [air1pm-avg.3:],
the result is:
2020-05-28,19:49,21.1,30,2.6,1025.0,1.1,3.6,100,0.0, 33.50, 5.0, 4.000, 22.4, 28, 39, 46, 20
2020-05-28,19:50,21.1,30,2.6,1025.0,0.9,2.6,93,0.0, 33.50, 5.0, 4.000, 22.4, 28, 39, 46, 20
which shows that my interpretation seems to be correct.

I think from what I read the difference is between avg1 and avg.1 - the "." is the formatting and the number behind is the decimal place
whereas (as per wiki) avg1, avg2, avg3, ..., avg60: selects average value from the last one to 60 minutes

Anyhow, my definition works, and there was no correction by Boris after having asked for.
The only thing which does not work is replacing avg by act.

I experimented with -act, -act1, -act.1, -avg1: result Blank
2020-05-28,19:49,21.1,30,2.6,1025.0,1.1,3.6,100,0.0, 33.50, , , 22.4, 28, 39, 46, 20
2020-05-28,19:50,21.1,30,2.6,1025.0,0.9,2.6,93,0.0, 33.50, , , 22.4, 28, 39, 46, 20

only avg1.1 and -avg.1 bring the same values with one digit/place after the decimal point
2020-05-28,19:49,21.1,30,2.6,1025.0,1.1,3.6,100,0.0, 33.50, 5.0, 4.0, 22.4, 28, 39, 46, 20
2020-05-28,19:50,21.1,30,2.6,1025.0,0.9,2.6,93,0.0, 33.50, 5.0, 4.0, 22.4, 28, 39, 46, 20

So my export and its data definition provide (at least at first glance) the correct results.
The only thing I'm wondering about is why -act (in whichever formatting notation) only brings a blank value.

Therefore, and because with the old firmware no more gaps occurred so far, I doubt that the gaps result from a wrong data definition of my "script".
« Last Edit: May 28, 2020, 03:03:08 PM by Gyvate »
WS2350 1.6.7, GW1000(3) 1.7.7,WH2650 WiFi (2) 1.7.7 (test/backup), GW1100 2.3.1, GW2000(3) 3.1.1, HP2551 1.9.5,5.1.5;HP3500 1.7.2,WS3800 1.2.8, WN1910 1.2.3,WN1980 1.2.3;
Ecowitt WS90(2)1.3.5/1.4.0, WS80(2)1.2.5, WS68, WS69, WH40, WH31, WH31-EP, WN30, WN34L, WN35, WH32, WH32-EP, WH32B, WH57 [Lightning], WH41 [PM2.5], WH51, WH45, WH55
MeteobridgePro(2)[test,prod] 5.8 Mar 01 2024, 15185 - Blake-Larsen Sun Recorder - RPi4/weewx 4.8.0/4.10.2/CumulusMX 3283/Meteobridge RPi4B-2GB(3169)
Barani Meteoshield Pro, MetSpec Rad02 - Ecowitt 5763,34418;WU ISAARB3(WH4000SE),ISAARB22(HP2553), http://meshka.eu

Offline galfert

  • Global Moderator
  • Forecaster
  • *****
  • Posts: 6822
Here is my recommendation for proper formatting. The numbers after that decimal should be zeros as they are just place holders for number of significant digits. There is never a .1 in that case. It would be .0 or .00 or .000 depending on data type. For example you'd want .00 for humidity because that is two digits, and you'd want .000 for wind direction because that is 3 digits and it then strips decimal from wind direction degrees. Likewise it strips the decimal from humidity since the sensor only reports in whole percentage...but it you leave humidity alone without modification (same with wind direction) then you end up with .0 for those datum.

[YYYY]-[MM]-[DD],[hh]:[mm],[th0temp-avg1],[th0hum-avg1.00],[th0dew-avg1],[thb0seapress-avg1],[wind0avgwind-avg1],[wind0wind-max1],[wind0dir-avg1.000],[rain0total-daysum], [sol0rad-avg1], [air0pm-avg1], [air1pm-avg1], [th1temp-avg1], [th1hum-avg1.00], [th20hum-avg.00], [th21hum-avg1.00], [th22hum-avg1.00]
« Last Edit: May 28, 2020, 03:26:45 PM by galfert »
Ecowitt GW1000 | Meteobridge on Raspberry Pi
WU: KFLWINTE111  |  PWSweather: KFLWINTE111
CWOP: FW3708  |  AWEKAS: 14814
Windy: pws-f075acbe
Weather Underground Issue Tracking
Tele-Pole

Offline galfert

  • Global Moderator
  • Forecaster
  • *****
  • Posts: 6822
I see that today Boris updated firmware to include the following (adding air export)???:

Quote
released May 28, 2020

adds so far missing sensor types "air" and "lgt" to data export function.

I'm not sure what he did differently that wasn't already supported before. I've been exporting my Ecowitt WH41 PM2.5 as air-pm for many months. I even wrote up a neat mySQL query to prevent situations of no write when air0pm was not available as tends to happen when it needs manual charging.
https://www.wxforum.net/index.php?topic=37851.0

« Last Edit: May 28, 2020, 03:44:23 PM by galfert »
Ecowitt GW1000 | Meteobridge on Raspberry Pi
WU: KFLWINTE111  |  PWSweather: KFLWINTE111
CWOP: FW3708  |  AWEKAS: 14814
Windy: pws-f075acbe
Weather Underground Issue Tracking
Tele-Pole

Offline Gyvate

  • Forecaster
  • *****
  • Posts: 3329
Here is my recommendation for proper formatting. The numbers after that decimal should be zeros as they are just place holders for number of significant digits. There is never a .1 in that case. It would be .0 or .00 or .000 depending on data type. For example you'd want .00 for humidity because that is two digits, and you'd want .000 for wind direction because that is 3 digits and it then strips decimal from wind direction degrees. Likewise it strips the decimal from humidity since the sensor only reports in whole percentage...but it you leave humidity alone without modification (same with wind direction) then you end up with .0 for those datum.

[YYYY]-[MM]-[DD],[hh]:[mm],[th0temp-avg1],[th0hum-avg1.00],[th0dew-avg1],[thb0seapress-avg1],[wind0avgwind-avg1],[wind0wind-max1],[wind0dir-avg1.000],[rain0total-daysum], [sol0rad-avg1], [air0pm-avg1], [air1pm-avg1], [th1temp-avg1], [th1hum-avg1.00], [th20hum-avg.00], [th21hum-avg1.00], [th22hum-avg1.00]

Thanks for providing your definition - AND
Please do not kill/blame the messenger

I used/copied your definition and after adding the missing colons (":") the result was:

[YYYY]-[MM]-[DD],[hh]:[mm],[th0temp-avg1:],[th0hum-avg1.00:],[th0dew-avg1:],[thb0seapress-avg1:],[wind0avgwind-avg1:],[wind0wind-max1:],[wind0dir-avg1.000:],[rain0total-daysum:], [sol0rad-avg1:], [air0pm-avg1:], [air1pm-avg1:], [th1temp-avg1:], [th1hum-avg1.00:], [th20hum-avg.00:], [th21hum-avg1.00:], [th22hum-avg1.00:]

2020-05-28,20:00,,,,,,,,0.0, , , , , , 39, ,
2020-05-28,20:01,,,,,,,,0.0, , , , , , 39, ,
2020-05-28,20:02,,,,,,,,0.0, , , , , , 39, ,
2020-05-28,20:03,,,,,,,,0.0, , , , , , 39, ,
2020-05-28,20:04,,,,,,,,0.0, , , , , , 39, ,
2020-05-28,20:05,,,,,,,,0.0, , , , , , 39, ,
2020-05-28,20:06,,,,,,,,0.0, , , , , , 39, ,

Interestingly (not sure if on purpose) you have left the definition of the 2nd soil moisture sensor as -avg.0 whereas almost everything else had avg1.x.
That's the only one that shows with a proper value.  :roll:

whereas my definition provides:
[YYYY]-[MM]-[DD],[hh]:[mm],[th0temp-avg.1:],[th0hum-avg.0:],[th0dew-avg.1:],[thb0seapress-avg.1:],[wind0avgwind-avg.1:],[wind0wind-max.1:],[wind0dir-avg.0:],[rain0total-sum.1:], [sol0rad-avg.2:], [air0pm-avg.1:], [air1pm-avg.1:], [th1temp-avg.1:], [th1hum-avg.0:], [th20hum-avg.0:], [th21hum-avg.0:], [th22hum-avg.0:]

2020-05-28,20:00,20.9,30,2.4,1025.0,1.4,3.6,87,0.0, 31.00, 5.0, 5.0, 22.4, 28, 39, 46, 20
2020-05-28,20:01,20.8,29,2.1,1025.1,0.7,2.0,83,0.0, 30.00, 5.0, 5.0, 22.4, 28, 39, 75, 54
2020-05-28,20:02,20.8,29,2.1,1024.9,1.0,2.0,93,0.0, 30.50, 5.0, 5.0, 22.3, 27, 39, 59, 52
2020-05-28,20:03,20.7,29,2.0,1024.9,0.6,1.5,111,0.0, 29.50, 5.0, 5.0, 22.3, 27, 39, 54, 51
2020-05-28,20:04,20.7,29,2.0,1025.0,0.3,1.5,92,0.0, 29.50, 5.0, 5.0, 22.3, 27, 39, 51, 50

What am I now supposed to make out of that ? :shock:
Do I have a crazy MB Pro ?

By the way - from a practical point of view I am happy with the results - finally a complete minute based export (even though the air0pm and air1pm values didn't export on the hourly or daily resolution either). And for the gaps (the main reason for this thread) we will see in about 36 hours.
But my curiosity has been kindled by the fact that these -act values do not work for me/with me how they should.
« Last Edit: May 28, 2020, 03:51:23 PM by Gyvate »
WS2350 1.6.7, GW1000(3) 1.7.7,WH2650 WiFi (2) 1.7.7 (test/backup), GW1100 2.3.1, GW2000(3) 3.1.1, HP2551 1.9.5,5.1.5;HP3500 1.7.2,WS3800 1.2.8, WN1910 1.2.3,WN1980 1.2.3;
Ecowitt WS90(2)1.3.5/1.4.0, WS80(2)1.2.5, WS68, WS69, WH40, WH31, WH31-EP, WN30, WN34L, WN35, WH32, WH32-EP, WH32B, WH57 [Lightning], WH41 [PM2.5], WH51, WH45, WH55
MeteobridgePro(2)[test,prod] 5.8 Mar 01 2024, 15185 - Blake-Larsen Sun Recorder - RPi4/weewx 4.8.0/4.10.2/CumulusMX 3283/Meteobridge RPi4B-2GB(3169)
Barani Meteoshield Pro, MetSpec Rad02 - Ecowitt 5763,34418;WU ISAARB3(WH4000SE),ISAARB22(HP2553), http://meshka.eu

Offline galfert

  • Global Moderator
  • Forecaster
  • *****
  • Posts: 6822
Well there are some differences in your implementation and mine. You are using a script. I'm just using a quick and dirty one-time alarm alert to email. I'm using the Test button to manually trigger. My format produces data. I threw in the missing colon : and I still get data. It doesn't seem to matter one way or the other to leave it out or not.

My Meteobridge is a standard not Pro. My Meteobridge is running today's latest firmware 4.3 build 13497.

 [ You are not allowed to view attachments ]
« Last Edit: May 28, 2020, 03:58:40 PM by galfert »
Ecowitt GW1000 | Meteobridge on Raspberry Pi
WU: KFLWINTE111  |  PWSweather: KFLWINTE111
CWOP: FW3708  |  AWEKAS: 14814
Windy: pws-f075acbe
Weather Underground Issue Tracking
Tele-Pole

Offline Gyvate

  • Forecaster
  • *****
  • Posts: 3329
Well there are some differences in your implementation and mine. You are using a script. I'm just using a quick and dirty one-time alarm alert to email. I'm using the Test button to manually trigger. My format produces data. I threw in the missing colon : and I still get data. It doesn't seem to matter one way or the other to leave it out or not.

My Meteobridge is a standard no Pro. My Meteobridge is running today's latest firmware 4.3 build 13497.

In the Pro it makes a difference. It has a clear formatting purpose.
So the colon inside the brackets obviously defines the end of the variable's data definition.

The output from your 1:1 copy with only one colon in the date/time definition was:

2020-05-28,20:00,,[th0hum-avg1.00],[th0dew-avg1],[thb0seapress-avg1],[wind0avgwind-avg1],[wind0wind-max1],[wind0dir-avg1.000],0.0, [sol0rad-avg1], [air0pm-avg1], [air1pm-avg1], [th1temp-avg1], [th1hum-avg1.00], 39, [th21hum-avg1.00], [th22hum-avg1.00]
2020-05-28,20:01,,[th0hum-avg1.00],[th0dew-avg1],[thb0seapress-avg1],[wind0avgwind-avg1],[wind0wind-max1],[wind0dir-avg1.000],0.0, [sol0rad-avg1], [air0pm-avg1], [air1pm-avg1], [th1temp-avg1], [th1hum-avg1.00], 39, [th21hum-avg1.00], [th22hum-avg1.00]
2020-05-28,20:02,,[th0hum-avg1.00],[th0dew-avg1],[thb0seapress-avg1],[wind0avgwind-avg1],[wind0wind-max1],[wind0dir-avg1.000],0.0, [sol0rad-avg1], [air0pm-avg1], [air1pm-avg1], [th1temp-avg1], [th1hum-avg1.00], 39, [th21hum-avg1.00], [th22hum-avg1.00]
2020-05-28,20:03,,[th0hum-avg1.00],[th0dew-avg1],[thb0seapress-avg1],[wind0avgwind-avg1],[wind0wind-max1],[wind0dir-avg1.000],0.0, [sol0rad-avg1], [air0pm-avg1], [air1pm-avg1], [th1temp-avg1], [th1hum-avg1.00], 39, [th21hum-avg1.00], [th22hum-avg1.00]
2020-05-28,20:04,,[th0hum-avg1.00],[th0dew-avg1],[thb0seapress-avg1],[wind0avgwind-avg1],[wind0wind-max1],[wind0dir-avg1.000],0.0, [sol0rad-avg1], [air0pm-avg1], [air1pm-avg1], [th1temp-avg1], [th1hum-avg1.00], 39, [th21hum-avg1.00], [th22hum-avg1.00]

and we are running the same software level Meteobridge 4.3 (May 28 2020, build 13497). Firmware version 1.4 obviously applies to the device, which in my case is a smartbedded device whereas yours is probably some re-purposed router for which the firmware used may be different. I say "may" because I don't know. I'd expect any firmware to be device specific, but maybe these routers which can be re-purposed for use with Meteobridge happen to have similar components which can be managed by the same piece of firmware.

But for figuring out if the gaps occur with GW1000 FW 1.5.9 in your setup or not this shouldn't be relevant.
If by the end of tomorrow there are no gaps with FW 1.5.6, I will set back to the device with 1.5.9 and see, if the phenomenon is still there.
If yes, I'll probably contact Ecowitt with my symptom description.
« Last Edit: May 29, 2020, 05:53:20 AM by Gyvate »
WS2350 1.6.7, GW1000(3) 1.7.7,WH2650 WiFi (2) 1.7.7 (test/backup), GW1100 2.3.1, GW2000(3) 3.1.1, HP2551 1.9.5,5.1.5;HP3500 1.7.2,WS3800 1.2.8, WN1910 1.2.3,WN1980 1.2.3;
Ecowitt WS90(2)1.3.5/1.4.0, WS80(2)1.2.5, WS68, WS69, WH40, WH31, WH31-EP, WN30, WN34L, WN35, WH32, WH32-EP, WH32B, WH57 [Lightning], WH41 [PM2.5], WH51, WH45, WH55
MeteobridgePro(2)[test,prod] 5.8 Mar 01 2024, 15185 - Blake-Larsen Sun Recorder - RPi4/weewx 4.8.0/4.10.2/CumulusMX 3283/Meteobridge RPi4B-2GB(3169)
Barani Meteoshield Pro, MetSpec Rad02 - Ecowitt 5763,34418;WU ISAARB3(WH4000SE),ISAARB22(HP2553), http://meshka.eu

Offline Gyvate

  • Forecaster
  • *****
  • Posts: 3329
update after 18 hours with GW1000 FW 1.5.6 - no gaps
when the 24 hours are complete I will fall back to the device with FW 1.5.9 and see if the earlier results remain
(or if the MB software update yesterday made a change).
« Last Edit: May 29, 2020, 06:01:19 AM by Gyvate »
WS2350 1.6.7, GW1000(3) 1.7.7,WH2650 WiFi (2) 1.7.7 (test/backup), GW1100 2.3.1, GW2000(3) 3.1.1, HP2551 1.9.5,5.1.5;HP3500 1.7.2,WS3800 1.2.8, WN1910 1.2.3,WN1980 1.2.3;
Ecowitt WS90(2)1.3.5/1.4.0, WS80(2)1.2.5, WS68, WS69, WH40, WH31, WH31-EP, WN30, WN34L, WN35, WH32, WH32-EP, WH32B, WH57 [Lightning], WH41 [PM2.5], WH51, WH45, WH55
MeteobridgePro(2)[test,prod] 5.8 Mar 01 2024, 15185 - Blake-Larsen Sun Recorder - RPi4/weewx 4.8.0/4.10.2/CumulusMX 3283/Meteobridge RPi4B-2GB(3169)
Barani Meteoshield Pro, MetSpec Rad02 - Ecowitt 5763,34418;WU ISAARB3(WH4000SE),ISAARB22(HP2553), http://meshka.eu

Offline ConligWX

  • Forecaster
  • *****
  • Posts: 843
  • #conligwx
    • conligwx.org
I see another MB update today, though looks like one user already has problems with it.  set your MB not to update on boot just yet.

https://forum.meteohub.de/viewtopic.php?f=61&t=14881
Regards Simon
Davis Vantage Pro2 Plus (6162UK) • Daytime FARS • WeatherLink Live • AirLink • PurpleAir PA-II-SD • CumulusMX •


Offline galfert

  • Global Moderator
  • Forecaster
  • *****
  • Posts: 6822
There are now several users reporting gap issues with firmware 1.5.9 when using a Meteobridge. Boris is actively working on this issue. Firmware 1.5.9 brought some big changes to support the not yet released soil temperature and moisture sensors. At this point we will need to just wait for Boris to update Meteobridge to support this newest firmware from Ecowitt. Hopefully the problem is not on Ecowitt's side in the reliability of the API to consistently keep providing live data. Ecowitt is also aware of the issue but they have not yet determined if the problem is on their side. Just my gut feeling is that the Meteobridge has to be updated to support the newest features of the latest GW1000 firmware. And yes Boris has the latest GW1000 API documentation for version 1.5.9.
Ecowitt GW1000 | Meteobridge on Raspberry Pi
WU: KFLWINTE111  |  PWSweather: KFLWINTE111
CWOP: FW3708  |  AWEKAS: 14814
Windy: pws-f075acbe
Weather Underground Issue Tracking
Tele-Pole

Offline Gyvate

  • Forecaster
  • *****
  • Posts: 3329
a relief to see/hear/read that's not my script but something else - still to be determined if on the Meteobridge side or on the Ecowitt side.

Boris' 2nd latest version from 28-May-2020 reads the live data properly - also for my two PM2.5 and three Soil Moisture sensors. Their values are in the database (both RAM and Disk [what Boris calls Archive]). I have followed ConligWX's advice and set my MB Pro not to update at reboot but keep the current version as there are obviously other issues I don't want to be exposed to.
@ConligWX: Thanks for the hint!  =D>

Boris checked on my device yesterday (for the PM.5 sensor issue - not shown in graph and not able to export: solved now).

After the upgrade to  build 13497 and Boris' visit on my device, my databases (RAM and Disk) crashed somehow - I suddenly had two empty databases.
Restore from backup left me with a one day gap which I have meanwhile manually restored from my WH4000SE database and the extra sensors from Ecowitt.net. As a lesson learnt I have now implemented a 15 minute backup cycle (as compared to daily before) to the internal disk and parallel copy to my PC and my NAS.

Also, all my own scripts, templates etc. had disappeared from the ../data/ directories (luckily also saved to the PC).
Did anyone observe this behaviour too ?
After a reboot or cold start I had already once before observed that the ../data directory was purged and only the files which come with the installation copied there again newly.
« Last Edit: May 29, 2020, 12:30:40 PM by Gyvate »
WS2350 1.6.7, GW1000(3) 1.7.7,WH2650 WiFi (2) 1.7.7 (test/backup), GW1100 2.3.1, GW2000(3) 3.1.1, HP2551 1.9.5,5.1.5;HP3500 1.7.2,WS3800 1.2.8, WN1910 1.2.3,WN1980 1.2.3;
Ecowitt WS90(2)1.3.5/1.4.0, WS80(2)1.2.5, WS68, WS69, WH40, WH31, WH31-EP, WN30, WN34L, WN35, WH32, WH32-EP, WH32B, WH57 [Lightning], WH41 [PM2.5], WH51, WH45, WH55
MeteobridgePro(2)[test,prod] 5.8 Mar 01 2024, 15185 - Blake-Larsen Sun Recorder - RPi4/weewx 4.8.0/4.10.2/CumulusMX 3283/Meteobridge RPi4B-2GB(3169)
Barani Meteoshield Pro, MetSpec Rad02 - Ecowitt 5763,34418;WU ISAARB3(WH4000SE),ISAARB22(HP2553), http://meshka.eu

Offline Gyvate

  • Forecaster
  • *****
  • Posts: 3329
After 72 hours of observations no gaps occur in the exports anymore - based on FW 1.5.6 of GW1000
With FW 1.5.9 the gaps remain (5 min, 7-8 min within an hour window, but not in constant time intervals [i.e. time between occurrences],  once even one of 1 1/2 h).
I have opened a "ticket" (thread) in the Meteohub forum, Meteobridge Pro section, for documentation and as a reminder and reference for Boris.
The fact that gaps occur only in the exports (and in the meteobridge database) but the data from the GW1000 are visible in ecowitt.net for the time slots of the gaps make it likely that the issue is not (only) with Ecowitt but rather with the Meteobridge software. FW 1.5.9 has a changed API afaik.
WS2350 1.6.7, GW1000(3) 1.7.7,WH2650 WiFi (2) 1.7.7 (test/backup), GW1100 2.3.1, GW2000(3) 3.1.1, HP2551 1.9.5,5.1.5;HP3500 1.7.2,WS3800 1.2.8, WN1910 1.2.3,WN1980 1.2.3;
Ecowitt WS90(2)1.3.5/1.4.0, WS80(2)1.2.5, WS68, WS69, WH40, WH31, WH31-EP, WN30, WN34L, WN35, WH32, WH32-EP, WH32B, WH57 [Lightning], WH41 [PM2.5], WH51, WH45, WH55
MeteobridgePro(2)[test,prod] 5.8 Mar 01 2024, 15185 - Blake-Larsen Sun Recorder - RPi4/weewx 4.8.0/4.10.2/CumulusMX 3283/Meteobridge RPi4B-2GB(3169)
Barani Meteoshield Pro, MetSpec Rad02 - Ecowitt 5763,34418;WU ISAARB3(WH4000SE),ISAARB22(HP2553), http://meshka.eu

 

anything