Author Topic: MB Pro extra sensor data upload to Meteotemplate incomplete - workaround  (Read 1732 times)

0 Members and 1 Guest are viewing this topic.

Offline Gyvate

  • Forecaster
  • *****
  • Posts: 1786
OK - you have three stations and station 1 and 2 provide three air quality sensors each.
These pictures would have been helpful from the start.

then try:
[YYYY]-[MM]-[DD],[hh]:[mm],[th0!0temp-lastval.1:],[th0!0hum-lastval.0:],[air1!0pm-lastval.1:],[air1!1pm-lastval.1:],[air1!2pm-lastval.1:],[air2!0pm-lastval.1:],[air2!1pm-lastval.1:],[air2!2pm-lastval.1:]

time frame: yesterday - resolution: minute (don't forget to save the event before pressing the "test" button)

another test option (same time frame, same resolution) could be:
[YYYY]-[MM]-[DD],[hh]:[mm],[th0!0temp-avg.1:],[th0!0hum-avg.0:],[air1!0pm-avg.1:],[air1!1pm-avg.1:],[air1!2pm-avg.1:],[air2!0pm-avg.1:],[air2!1pm-avg.1:],[air2!2pm-avg.1:]
« Last Edit: May 17, 2022, 07:37:31 PM by Gyvate »
WS2350 1.6.4, GW1000(3) 1.7.4, WH2650 WiFi (2) 1.7.4 (test/backup), GW1100 2.1.7, GW2000(3) 2.1.7, HP2551 1.6.4/1.8.1;
Ecowitt WS90(2), WS80, WH40, WH65, WH31(5), WH31-EP, WN30, WH32, WH32-EP, WH32B, WH57 [Lightning] (1), WH41 [PM2.5] (2), WH51 (5), WH45, WH55 (2)
MeteobridgePro(2)[test,prod 5.4 Aug 2 2022, 14798 - Blake-Larsen Sun Recorder - RPi4/weewx 4.8.0/CumulusMX 3196/MB RPi4(2826)
Barani Meteoshield Pro, MetSpec Rad02 - Ecowitt 5763,34418;WU ISAARB3(WH4000SE), ISAARB22(HP2553), http://meshka.eu

Offline alexvanuxem

  • Member
  • *
  • Posts: 22
thx, the last try with the avg.1 worked,
i get my 6 pp's exported

getting somewhere finally, i thought then it was solved , but,

when i put it into the string:

api.php?U=[epoch.1:]&T=[th0temp-lastval.1:]&TMX=[th0temp-max5:]&TMN=[th0temp-min5:]&H=[th0hum-lastval.1:]&P=[thb0seapress-lastval:1]&W=[wind0avgwind-lastval=kmh:1]&G=[wind0wind-lastval=kmh:1]&B=[wind0dir-lastval:1]&R=[rain0total-daysum:--]&RR=[rain0rate-lastval:1]&S=[sol0rad-lastval:2]&UV=[uv0index-lastval:1]&TIN=[thb0temp-lastval.1:]&HIN=[thb0hum-lastval.1:]&T1=[th1temp-lastval.1:]&H1=[th1hum-lastval.1:]&SM1=[th20hum-lastval.0:]&SM2=[th21hum-lastval.0:]&SM3=[th22hum-lastval.0:]&SM4=[th23hum-lastval.0:]&SM5=[th24hum-lastval.0:]&L=[lgt0energy-lastval.0:]& PP1=[air1!0pm-avg.1:]&PP2=[air1!1pm-avg.1:]&PP3=[air1!2pm-avg.1:]&PP4=[air2!0pm-avg.1:]&PP5=[air2!1pm-avg.1:]&PP6=[air2!2pm-avg.1:]& PASS=(my password)


i don't get it in meteo template

see picture

at least we, I, made progress, thx again

Offline Gyvate

  • Forecaster
  • *****
  • Posts: 1786
when MB posts the expression instead of a value, it is because it doesn't find anything in the database.
if you had put [air2!0pm-avg.1:--], it would send the -- instead.

Your posting to MT is in which timing ? (10 seconds, 30 seconds, 1 minute, 5 minutes ...) ?

I think it has to do with the timing the air sensors of station 2 produce new data and this data is stored in the database.
You could try [air2!0pm-lastval.1:--] instead (for the 2! AQ sensors only) or  [air2!0pm-act.1:--]

Do you have minute based sensor data archiving activated in the System --> Administration Tab, Database section ?

are these station 2 air quality sensors different hardware from the station 1 air quality sensors ?

You can see that the syntax is correct as the values of the other sensors and air sensors are sent !

What Meteobridge hardware do you use in the first place ? MB Pro, MB Nano, MB RPi ?
What's your (=MB) RAM usage ?

what does MB report from the database  (History, edit minute data) for the sensors air2!xpm ? (x=0,1,2) for the actual time ?
« Last Edit: May 18, 2022, 02:23:43 AM by Gyvate »
WS2350 1.6.4, GW1000(3) 1.7.4, WH2650 WiFi (2) 1.7.4 (test/backup), GW1100 2.1.7, GW2000(3) 2.1.7, HP2551 1.6.4/1.8.1;
Ecowitt WS90(2), WS80, WH40, WH65, WH31(5), WH31-EP, WN30, WH32, WH32-EP, WH32B, WH57 [Lightning] (1), WH41 [PM2.5] (2), WH51 (5), WH45, WH55 (2)
MeteobridgePro(2)[test,prod 5.4 Aug 2 2022, 14798 - Blake-Larsen Sun Recorder - RPi4/weewx 4.8.0/CumulusMX 3196/MB RPi4(2826)
Barani Meteoshield Pro, MetSpec Rad02 - Ecowitt 5763,34418;WU ISAARB3(WH4000SE), ISAARB22(HP2553), http://meshka.eu

Offline alexvanuxem

  • Member
  • *
  • Posts: 22
made some progress , when i add
[air0pm-lastval:2]
[air1pm-lastval:2]
[air2pm-lastval:2]
[air2!0pm-lastval:2]
[air2!1pm-lastval:2]
[air2!2pm-lastval:2]
--
[air2!0pm-lastval.1:--]
[air2!0pm-act.1:--]

into a realtimeMB template and upload it, i get all the data.

but not into MT :-(

the fact that senor 1 is visible, is because it's mapped inside MB

the storage is persistent inside MT,

i also get through my wetaherlink live.

i think it's on the mt side,
thx again for your help.

Offline davidefa

  • Forecaster
  • *****
  • Posts: 359
Extract from your ( meteotemplate ) cache/apiLog.txt:
Code: [Select]
2022-05-18 13:29:15
INFOS:
Authorized access.
Current date/time: 2022-05-18 13:29:15
api called by http
update U: 1652873354
update SW: meteobridge
update T: 25.9
update TMX: 25.9
update TMN: 25.9
update H: 48
update P: 1022.3
update W: 6.1
update G: 7.9
update B: 230
update R: 0.0
update RR: 0.0
update S: 821.0
update UV: 5.1
update TIN: 24.1
update HIN: 50
update TS1: 46.5
update TS2: 30.8
update TS3: 20.7
update TS4: 37.2
update SM1: 0.0
update SM2: 0.0
update SM3: 0.0
update SM4: 0.0
update SM5: 0.0
update SM6: 0.0
update LW1: 0.0
update LT1: 46.5
update LW2: 0.0
update LT2: 30.8
update PP3: 5.1
update PP2: 8.4
update PP1: 9.1
Begin of common part of api script
Start handling live data
Checking if extra sensors should be logged
Extra sensors data to save in db: TIN, HIN, UV, ET, TS1, LW1, LT1, SM1, PP1, TS2, LW2, LT2, SM2, PP2, TS3, SM3, PP3, TS4, SM4, SM5, SM6
add live data: D: 14
add live data: DTime: 1652873344
add live data: A: 26.3
add live data: ATime: 1652873344
calculated D: 14
calculated A: 26
Save meteotemplateLive.txt: {"U":1652873354,"SW":"meteobridge","T":25.9,"TMX":25.9,"TMN":25.9,"H":48,"P":1022.3,"W":6.1,"G":7.9,"B":230,"R":0,"RR":0,"S":821,"UV":5.1,"TIN":24.1,"HIN":50,"TS1":46.5,"TS2":30.8,"TS3":20.7,"TS4":37.2,"SM1":0,"SM2":0,"SM3":0,"SM4":0,"SM5":0,"SM6":0,"LW1":0,"LT1":46.5,"LW2":0,"LT2":30.8,"PP3":5.1,"PP2":8.4,"PP1":9.1,"UTime":1652873355,"SWTime":1652873355,"TTime":1652873355,"TMXTime":1652873355,"TMNTime":1652873355,"HTime":1652873355,"PTime":1652873355,"WTime":1652873355,"GTime":1652873355,"BTime":1652873355,"RTime":1652873355,"RRTime":1652873355,"STime":1652873355,"UVTime":1652873355,"TINTime":1652873355,"HINTime":1652873355,"TS1Time":1652873355,"TS2Time":1652873355,"TS3Time":1652873355,"TS4Time":1652873355,"SM1Time":1652873355,"SM2Time":1652873355,"SM3Time":1652873355,"SM4Time":1652873355,"SM5Time":1652873355,"SM6Time":1652873355,"LW1Time":1652873355,"LT1Time":1652873355,"LW2Time":1652873355,"LT2Time":1652873355,"PP3Time":1652873355,"PP2Time":1652873355,"PP1Time":1652873355,"D":14,"DTime":1652873355,"A":26,"ATime":1652873355}

The first part with the list of:
Code: [Select]
update SSS: XXXis the list of the parameters passed to api.php ( you can pass any parameter for example if you pass: &MIKEY=MOUSE youll find a row 'update MIKEY: MOUSE' )
from this list I can see there is no PP4, PP5, PP6

For a further check you can use the 'ear' tool ( from olicat ) this will show you what your meteobridge is sending
Simply direct your meteobridge upload to the ear tool

NOTE
When making these tests ( changing the fields sent to the api.php script ) you should/could delete the meteotemplateLive.txt file, not to get confused ( by new and 'stale' values/fields ).
I mean that even if you send a field only once ( the MIKEY=MOUSE in the above example ) the meteotemplateLive.txt file will contein this field for 30 minutes ( then it will be removed ).
This is, I think, to  overcome the wireless sensors 'hiccups' that may happen.
These 'stale' fields are distinguished in the cache/apiLog.txt file as:
Code: [Select]
add live data: MIKEY: MOUSE

Offline Gyvate

  • Forecaster
  • *****
  • Posts: 1786
can you post (attach !) the MT meteotemplateLive.txt from the .../template directory after the MB posts (it contains your MB posted data)!?
Also, can you post (a) picture(s) of the MT database schema !?
as @davidefa has shown, something is fishy ....
MT will only accept data via the API for existing database fields - and the PPx fields are in the extra table.
When the PP4-PP6 markers are not contained in the meteotemplateLive.txt file, even though they have been posted by MB, maybe the API doesn't identify them as fields to be filled and archived.

Have you switched off the standard MB posting for Meteotemplate in "Weather Nets" ? - put  "upload interval" to "no upload" and save (!!)
If you have two parallel posts, the API might get a hickup.
« Last Edit: May 18, 2022, 08:16:25 AM by Gyvate »
WS2350 1.6.4, GW1000(3) 1.7.4, WH2650 WiFi (2) 1.7.4 (test/backup), GW1100 2.1.7, GW2000(3) 2.1.7, HP2551 1.6.4/1.8.1;
Ecowitt WS90(2), WS80, WH40, WH65, WH31(5), WH31-EP, WN30, WH32, WH32-EP, WH32B, WH57 [Lightning] (1), WH41 [PM2.5] (2), WH51 (5), WH45, WH55 (2)
MeteobridgePro(2)[test,prod 5.4 Aug 2 2022, 14798 - Blake-Larsen Sun Recorder - RPi4/weewx 4.8.0/CumulusMX 3196/MB RPi4(2826)
Barani Meteoshield Pro, MetSpec Rad02 - Ecowitt 5763,34418;WU ISAARB3(WH4000SE), ISAARB22(HP2553), http://meshka.eu

Offline alexvanuxem

  • Member
  • *
  • Posts: 22
need some pointers for the eartool, does mb have a custom server option?

switched the upload to mt, data stops coming, switched on upload, data starts again;

see the requested files attached,  [ You are not allowed to view attachments ]  [ You are not allowed to view attachments ]

Offline davidefa

  • Forecaster
  • *****
  • Posts: 359
In meteotemplateLive.txt you will find any field api.php receives ( even the MIKEY field, if you pass it, which is clearly uknown to meteotemplate ), so if api.php 'GETS' the PP4-PP6 fields they will be in meteotemplateLive.txt
But these fields will be saved in the database only if enabled via the control panel ( but this is a second step, after you receive these fields correctly ).

Offline alexvanuxem

  • Member
  • *
  • Posts: 22
OK, thx.

the data is sent through the networks tab, you select meteotemplate, site name, password.

so if the template used by this, doesn't contain the datafields op pp, it isn't sent through, correct?

can you get yr hands on a template, which contains all the necessary fields, so I can upload this into MB, and send it through my site as meteotemplateLive.txt? just an open question.

thx for the clarification.


Offline Gyvate

  • Forecaster
  • *****
  • Posts: 1786
the data is either sent to the API by a predefined template (which is not directly accessible inside meteobridge - you would really have to do some deep digging), but this template is incomplete - hence the creation of the custom http request I has mentioned earlier.
The usual approach is to stop the posting via the "weather nets" tab and replace it by a custom event. (you cannot send data via the "Network" tab)

Theoretically, with some effort, you could create an export with the content of meteotemplateLive.txt (create an export named meteotemplateLive.exp or any name, rename it to meteotemplateLive.txt and send it via a ftp event (inside MB) to http://my-domain/my-web-root/template every let's say 15 seconds.
 
The API string starting after api.php? (without the password) starting with "{" and ending with "}" would be the first part of the data export definition (replacing the '&XXX='  by ',"XXX":' - without ' ' and a comma after the sensor selector expression)
e.g. {"U":[epoch.1:],"T":"[th0temp-lastval.1:]", etc.
and the second part would be
adding the '"XXXTime": "[epoch.1:]", for each marker variable (without ' ') - XXX = H, L, T, PP1, .... e.g. '"PP1Time":"[epoch.1:]",' (without ' ')

The result should look like:
Code: [Select]
{"U":1629299002,"T":20,"TMX":20.3,"TMN":19.9,"H":68,"P":1016,"W":3.2,"G":4,"B":118,"R":1.8,"RR":0,"S":219,"UV":1,"TIN":27,"HIN":50,"T1":21.5,"H1":63,"T2":24,"H2":72,"T3":20.2,"H3":67,"T4":20,"H4":73,"SM1":77,"SM2":71,"SM3":89,"SM4":27,"SM5":32,"L":0,"LD":14,"LT":1629113787,"PP1":6,"PP2":5,"PP3":0.8,"PP4":2,"CO2_1":792,"TS1":-22.3,"TS2":26.3,"TS3":6.9,"T5":-22.3,"T6":26.3,"H6":66,"T7":6.9,"H7":51,"T8":20.3,"H8":66,
"UTime":1629299103,"TTime":1629299103,"TMXTime":1629299103,"TMNTime":1629299103,"HTime":1629299103,"PTime":1629299103,"WTime":1629299103,"GTime":1629299103,"BTime":1629299103,"RTime":1629299103,"RRTime":1629299103,"STime":1629299103,"UVTime":1629299103,"TINTime":1629299103,"HINTime":1629299103,"T1Time":1629299103,"H1Time":1629299103,"T2Time":1629299103,"H2Time":1629299103,"T3Time":1629299103,"H3Time":1629299103,"T4Time":1629299103,"H4Time":1629299103,"SM1Time":1629299103,"SM2Time":1629299103,"SM3Time":1629299103,"SM4Time":1629299103,"SM5Time":1629299103,"LTime":1629299103,"LDTime":1629299103,"LTTime":1629299103,"PP1Time":1629299103,"PP2Time":1629299103,"PP3Time":1629299103,"PP4Time":1629299103,"CO2_1Time":1629299103,"TS1Time":1629299103,"TS2Time":1629299103,"TS3Time":1629299103,"T5Time":1629299103,"T6Time":1629299103,"H6Time":1629299103,"T7Time":1629299103,"H7Time":1629299103,"T8Time":1629299103,"H8Time":1629299103,"D":13.9,"DTime":1629299103,"A":20.6,"ATime":1629299103}
WS2350 1.6.4, GW1000(3) 1.7.4, WH2650 WiFi (2) 1.7.4 (test/backup), GW1100 2.1.7, GW2000(3) 2.1.7, HP2551 1.6.4/1.8.1;
Ecowitt WS90(2), WS80, WH40, WH65, WH31(5), WH31-EP, WN30, WH32, WH32-EP, WH32B, WH57 [Lightning] (1), WH41 [PM2.5] (2), WH51 (5), WH45, WH55 (2)
MeteobridgePro(2)[test,prod 5.4 Aug 2 2022, 14798 - Blake-Larsen Sun Recorder - RPi4/weewx 4.8.0/CumulusMX 3196/MB RPi4(2826)
Barani Meteoshield Pro, MetSpec Rad02 - Ecowitt 5763,34418;WU ISAARB3(WH4000SE), ISAARB22(HP2553), http://meshka.eu

Offline davidefa

  • Forecaster
  • *****
  • Posts: 359
Don't update a meteotemplateLive.txt file ( as doing this will not update the database ).
But you can upload a csv file ( a text file, with field separated by commas as you have already tested ).
Then configure meteotemplate ( control panel -> database update ) and select custom file and configure the fields you want to update.
You will have to setup 2 cron jobs ( the first to upload the csv file, the second to execute the update/update.php script, that will read the csv file and generate the api.php call with the configured fields )
Currently this method supports only the main sensors, but I can extend it to support also all extra sensors ( as needed by you ).

Offline alexvanuxem

  • Member
  • *
  • Posts: 22
thx a lot everybody!!

I will let it rest, for now, and take a look at it when I get some vacation, and time to try all this.
seems like a lot of work.

Learned a lot the last few days, so thank you again.


 

anything