Author Topic: Blitzortung back, but stations file structure changed  (Read 696 times)

0 Members and 1 Guest are viewing this topic.

Offline saratogaWX

  • Administrator
  • Forecaster
  • *****
  • Posts: 6819
  • Saratoga, CA, USA Weather - free PHP scripts
    • Saratoga-Weather.org
Blitzortung back, but stations file structure changed
« on: October 30, 2018, 12:34:00 AM »
Looks like the feed for the stations file has come back online.  The bad news is the new file has a different structure than the old file did, so the BO-mapgen script set will need some changes, which I'll get done on 30-Oct.

FYI .. old file looked like this
Code: [Select]
{
"station": "635",
"user": "787",
"city": "Peoria",
"comments": "Unshielded Ferrite Rods, 120 mm",
"country": "United States \/ Illinois",
"website": null,
"latitude": "40.819889",
"longitude": "-89.621334",
"altitude": "4",
"controller_board": "6.8",
"firmware": "WT 6.20.2 \/ 31e",
"controller_status": "10",
"input_0_board": null,
"input_1_board": null,
"input_2_board": null,
"input_3_board": null,
"input_4_board": null,
"input_5_board": null,
"input_0_firmware": null,
"input_1_firmware": null,
"input_2_firmware": null,
"input_3_firmware": null,
"input_4_firmware": null,
"input_5_firmware": null,
"input_0_gain": "0.0",
"input_1_gain": "0.0",
"input_2_gain": "7.7",
"input_3_gain": "7.7",
"input_4_gain": "7.7",
"input_5_gain": "7.7",
"input_0_antenna": "F;120;",
"input_1_antenna": "F;120;",
"input_2_antenna": ";;",
"input_3_antenna": ";;",
"input_4_antenna": ";;",
"input_5_antenna": ";;",
"signals": "0",
"last_signal": "2016-11-22 02:59:31",
"last_signal_nsec": "1479783571624764768",
"last_strike": "0000-00-00 00:00:00",
"last_strike_nsec": "0",
"50_km_60_m": "0",
"50_km_60_m_a": "0",
"50_km_60_m_e": "0",
"500_km_60_m": "0",
"500_km_60_m_a": "2",
"500_km_60_m_e": "0",
"5000_km_60_m": "0",
"5000_km_60_m_a": "1371",
"5000_km_60_m_e": "0"
}{
"station": "652",
"user": "787",
"city": "Peoria",
"comments": "Unshielded Ferrite Rods, 120 mm",
"country": "United States \/ Illinois",
"website": null,
"latitude": 29.4,
"longitude": -98.6,
"altitude": 0,
"controller_board": "6.8",
"firmware": "WT 6.20.2 \/ 30a",
"controller_status": "10",
"input_0_board": "5.7",
"input_1_board": "5.7",
"input_2_board": null,
"input_3_board": null,
"input_4_board": null,
"input_5_board": null,
"input_0_firmware": "30a",
"input_1_firmware": "30a",
"input_2_firmware": null,
"input_3_firmware": null,
"input_4_firmware": null,
"input_5_firmware": null,
"input_0_gain": "7.7",
"input_1_gain": "7.7",
"input_2_gain": "7.7",
"input_3_gain": "7.7",
"input_4_gain": "7.7",
"input_5_gain": "7.7",
"input_0_antenna": "F;250,7.5;",
"input_1_antenna": "F;250,7.5;",
"input_2_antenna": ";;",
"input_3_antenna": ";;",
"input_4_antenna": ";;",
"input_5_antenna": ";;",
"signals": "0",
"last_signal": "1970-01-01 00:00:01",
"last_signal_nsec": "1424470174",
"last_strike": "0000-00-00 00:00:00",
"last_strike_nsec": "0",
"50_km_60_m": "0",
"50_km_60_m_a": "0",
"50_km_60_m_e": "0",
"500_km_60_m": "0",
"500_km_60_m_a": "166",
"500_km_60_m_e": "0",
"5000_km_60_m": "0",
"5000_km_60_m_a": "1371",
"5000_km_60_m_e": "0"
} ...

New file looks like this
Code: [Select]
{
"1": {
"user": "2",
"city": null,
"comments": null,
"country": "Germany",
"website": "www.purecontest.de",
"latitude": 49.71988,
"longitude": 8.469994,
"altitude": 2,
"controller_board": "6.8",
"firmware": "WT 6.20.3 \/ 31e",
"controller_status": "10",
"input": [{
"board": "5.4",
"firmware": "31e",
"gain": "7.7",
"antenna": ";;"
}, {
"board": "5.4"
}, {
"board": "5.4"
}, {
"board": "5.4"
}, {
"board": "5.4"
}, {
"board": "5.4"
}
],
"signals": "0",
"last_signal": "2015-03-15 18:56:26",
"last_signal_nsec": "1426445786000000000",
"last_strike": "0000-00-00 00:00:00",
"last_strike_nsec": "0",
"50_km_60_m": "0",
"50_km_60_m_a": "0",
"50_km_60_m_e": "0",
"500_km_60_m": "0",
"500_km_60_m_a": "1",
"500_km_60_m_e": "0",
"5000_km_60_m": "0",
"5000_km_60_m_a": "5078",
"5000_km_60_m_e": "0"
},
"2": {
"user": "3",
"city": "Adelsdorf",
"comments": null,
"country": "Germany",
"website": "www.wetterstation-adelsdorf.de",
"latitude": 49.71038,
"longitude": 10.896905,
"altitude": 5,
"controller_board": "6.8",
"firmware": "WT 6.20.2 \/ 31e",
"controller_status": "30",
"input": [{
"board": "5.7",
"firmware": "31e",
"gain": "7.7",
"antenna": "F;120;S"
}, {
"board": "5.7"
}, {
"board": "5.7"
}, {
"board": "5.7"
}, {
"board": "5.7"
}, {
"board": "5.7"
}
],
"signals": "186",
"last_signal": "2018-08-28 13:41:15",
"last_signal_nsec": "1535463675864781343",
"last_strike": "2018-08-28 13:41:16",
"last_strike_nsec": "0",
"50_km_60_m": "0",
"50_km_60_m_a": "1",
"50_km_60_m_e": "0",
"500_km_60_m": "0",
"500_km_60_m_a": "1",
"500_km_60_m_e": "0",
"5000_km_60_m": "66",
"5000_km_60_m_a": "5079",
"5000_km_60_m_e": "0.0496351"
} ...
  Note the new file is keyed by Station number, instead of having a station in the record.

Grrr...  Both BO-mapgen-inc.php and BO-station-inc.php will need changes for the new format.
Ken True/Saratoga, CA, USA main site: saratoga-weather.org
Davis VP1+ FARS, Boltek-PCI/NexStorm, microSferics ToA, Blitzortung RED, GRLevel3, WD, WL, VWS, Cumulus, Meteobridge/hub
Free weather PHP scripts/website templates - update notifications on Twitter saratogaWXPHP

Offline Cutty Sark Sailor

  • WxElement panel
  • Forecaster
  • *****
  • Posts: 2966
    • Frankfort Weather - TwinHollies WeatherCenter
Re: Blitzortung back, but stations file structure changed
« Reply #1 on: October 30, 2018, 05:19:34 AM »
Ken....
Might sit on this a few more days,.. The current hardware and com issues aren't totally resolved yet.
Unrelated to the server hardware issue,
..there  will be some processing and presentation changes implemented, which may affect the scripts...

I suspect some access protocols may be modified.   :twisted: .. to thwart some folks who are violating access / usage restrictions.
 ...some of which is violating EU and other privacy laws with e.g.) operator and station location visible in the source codes they're using. :roll:
Hate to see you do a multiple re-write...
Mike
« Last Edit: October 30, 2018, 05:23:03 AM by Cutty Sark Sailor »
 

Offline saratogaWX

  • Administrator
  • Forecaster
  • *****
  • Posts: 6819
  • Saratoga, CA, USA Weather - free PHP scripts
    • Saratoga-Weather.org
Re: Blitzortung back, but stations file structure changed
« Reply #2 on: October 30, 2018, 01:42:40 PM »
S'Ok.. I got tired of seeing just strikes on the map.

I've attached the updated BO-mapgen-inc.php and BO-stations-inc.php to support the new stations file JSON.

I do notice that the stations file strike data for a station seems to not be updating in the JSON, so that's another thing for Egon to fix.

The V1.05 BO-mapgen-inc.php also grabs the en.blitzortung.org index page, parses for the 'message of the day' and saves that text into ./cache/BOmotd.txt file.

On my lightning pages, I now have a snippet of code to display that MOTD if there is one
Code: [Select]
<?php if(file_exists('./BOmap/cache/BOmotd.txt')) {
 $BOmotd file_get_contents('./BOmap/cache/BOmotd.txt');
 if(strlen($BOmotd) > 5) {
 print "<p style=\"text-align: center\">$BOmotd</p>\n";
 }
}
?>

Ken True/Saratoga, CA, USA main site: saratoga-weather.org
Davis VP1+ FARS, Boltek-PCI/NexStorm, microSferics ToA, Blitzortung RED, GRLevel3, WD, WL, VWS, Cumulus, Meteobridge/hub
Free weather PHP scripts/website templates - update notifications on Twitter saratogaWXPHP

Offline gwwilk

  • Southeast Lincoln Weather
  • Forecaster
  • *****
  • Posts: 2019
    • SouthEast Lincoln, NE Weather
Re: Blitzortung back, but stations file structure changed
« Reply #3 on: October 30, 2018, 02:14:15 PM »
Many thanks, Ken!  After running 'gen-BO-maps.php' on my testing server, the stations were back on the latest map.  Excellent work, as always. =D&gt; \:D/ =D&gt;
Regards, Jerry Wilkins
gwwilk@inebraska.com

Offline W0BTU

  • Member
  • *
  • Posts: 19
Re: Blitzortung back, but stations file structure changed
« Reply #4 on: October 30, 2018, 06:46:56 PM »
What does Egon mean here?
https://forum.blitzortung.org/showthread.php?tid=2759&pid=19561#pid19561
"...yes, this is a problem with our database, please reassign the station to your account...."

It sounds like he might be saying that the system is fixed as far as he can from his end, and now all participants need to do this.

If so, then exactly how do we do this?
Log in and delete our stations, and then set them back up again?
Stations: 1977, 2294

Offline gwwilk

  • Southeast Lincoln Weather
  • Forecaster
  • *****
  • Posts: 2019
    • SouthEast Lincoln, NE Weather
Re: Blitzortung back, but stations file structure changed
« Reply #5 on: October 30, 2018, 06:48:29 PM »
What does Egon mean here?
https://forum.blitzortung.org/showthread.php?tid=2759&pid=19561#pid19561
"...yes, this is a problem with our database, please reassign the station to your account...."

It sounds like he might be saying that the system is fixed as far as he can from his end, and now all participants need to do this.

If so, then exactly how do we do this?
Log in and delete our stations, and then set them back up again?
I think Egon means that only those who registered their stations within the past few weeks may need to re-register.  If your station isn't showing up, re-register.
Regards, Jerry Wilkins
gwwilk@inebraska.com

Offline Cutty Sark Sailor

  • WxElement panel
  • Forecaster
  • *****
  • Posts: 2966
    • Frankfort Weather - TwinHollies WeatherCenter
Re: Blitzortung back, but stations file structure changed
« Reply #6 on: October 30, 2018, 07:32:50 PM »
What does Egon mean here?
https://forum.blitzortung.org/showthread.php?tid=2759&pid=19561#pid19561
"...yes, this is a problem with our database, please reassign the station to your account...."

It sounds like he might be saying that the system is fixed as far as he can from his end, and now all participants need to do this.

If so, then exactly how do we do this?
Log in and delete our stations, and then set them back up again?
I think Egon means that only those who registered their stations within the past few weeks may need to re-register.  If your station isn't showing up, re-register.
Agreed... however, login first, see if station comes up under your operator account.  if it is listed under your account, just try re-entering the information, especially the computer ID.  Also make sure 'anonymous' is ticked "N". If you delete, or add a new station, you'll get a new station number...
Now, LightningMaps.org may still have your station listed as up and running, even though BT fails to show you. LMO and BT aren't speaking together well presently, so I don't know what this would do for that...
As Gerry said, this appears to only affect a very small number of newer, or previously 'improperly' configured registrations, and resulted in faulty 'new' db rollover.
 

Offline W0BTU

  • Member
  • *
  • Posts: 19
Re: Blitzortung back, but stations file structure changed
« Reply #7 on: October 30, 2018, 10:15:48 PM »
What does Egon mean here?
https://forum.blitzortung.org/showthread.php?tid=2759&pid=19561#pid19561
"...yes, this is a problem with our database, please reassign the station to your account...."

It sounds like he might be saying that the system is fixed as far as he can from his end, and now all participants need to do this.

If so, then exactly how do we do this?
Log in and delete our stations, and then set them back up again?
I think Egon means that only those who registered their stations within the past few weeks may need to re-register.  If your station isn't showing up, re-register.
Agreed... however, login first, see if station comes up under your operator account.  if it is listed under your account, just try re-entering the information, especially the computer ID.  Also make sure 'anonymous' is ticked "N". If you delete, or add a new station, you'll get a new station number...
Now, LightningMaps.org may still have your station listed as up and running, even though BT fails to show you. LMO and BT aren't speaking together well presently, so I don't know what this would do for that...
As Gerry said, this appears to only affect a very small number of newer, or previously 'improperly' configured registrations, and resulted in faulty 'new' db rollover.

My new station (2294) was brought online Sept 21, and perhaps I am in that "very small number of users". https://www.lightningmaps.org/blitzortung/america/index.php?bo_page=statistics&bo_show=longtime&lang=en&bo_station_id=1174.

I could try removing only 2294 and adding it again. I don't care if the station number changes.
« Last Edit: October 30, 2018, 10:29:32 PM by W0BTU »
Stations: 1977, 2294

Offline miraculon

  • Sunrise Side Weather
  • Forecaster
  • *****
  • Posts: 3208
  • KE8DAF
    • Sunrise Side Weather in Rogers City MI USA
Re: Blitzortung back, but stations file structure changed
« Reply #8 on: October 31, 2018, 08:43:54 AM »
Thanks, Ken. At first I was just happy to see the strikes again, but it is good to have the station status.

It is good to be able see intermittent issues with your station as the maps march along.

Greg H.


Blitzortung Stations #706 and #1682
CoCoRaHS: MI-PI-1
CWOP: CW4114 and KE8DAF-13
WU: KMIROGER7
Amateur Radio Callsign: KE8DAF

Offline PaulMy

  • Forecaster
  • *****
  • Posts: 4134
    • KomokaWeather
Re: Blitzortung back, but stations file structure changed
« Reply #9 on: October 31, 2018, 10:17:45 AM »
My station 1076 seems to be reporting http://www.komokaweather.com/BOmaps/indexON.php 

but getting an error in Last signal and Stroke count  https://www.lightningmaps.org/blitzortung/america/index.php?bo_page=statistics&bo_show=station&lang=en&bo_station_id=969


Do I need to do any changes from this new file structure?

I am still on firmware 8.0-b2 should I update?

Thanks,
Paul
Davis Vantage Pro 2 Plus 24-FARS Wireless
Envoy/USB DataLogger/Cumulus v.1.9.4   -   Vue Console/WiFiLogger
Komoka, ON  Canada
www.komokaweather.ca   www.komokaweather.com   www.komokaweather.com/weather28/   www.komokaweather.com/pws/index.php
Blitzortung Station #1076 www.lightningmaps.org/blitzortung7/america/index.php?bo_page=statistics&bo_show=network&lang=en&bo_station_id=969

Offline gwwilk

  • Southeast Lincoln Weather
  • Forecaster
  • *****
  • Posts: 2019
    • SouthEast Lincoln, NE Weather
Re: Blitzortung back, but stations file structure changed
« Reply #10 on: October 31, 2018, 10:36:11 AM »
My station 1076 seems to be reporting http://www.komokaweather.com/BOmaps/indexON.php 

but getting an error in Last signal and Stroke count  https://www.lightningmaps.org/blitzortung/america/index.php?bo_page=statistics&bo_show=station&lang=en&bo_station_id=969


Do I need to do any changes from this new file structure?

I am still on firmware 8.0-b2 should I update?

Thanks,
Paul
This is the same thing I'm seeing for my station at this time.  This almost certainly reflects the transitional state of the BO servers, and doesn't look like something we can fix.  We just need to be patient. :-)
Regards, Jerry Wilkins
gwwilk@inebraska.com

Offline saratogaWX

  • Administrator
  • Forecaster
  • *****
  • Posts: 6819
  • Saratoga, CA, USA Weather - free PHP scripts
    • Saratoga-Weather.org
Re: Blitzortung back, but stations file structure changed
« Reply #11 on: October 31, 2018, 11:02:11 AM »
The stations file seems to have 'stuck' statistics for stations .. only the timestamp seems to change.  I think it's an update issue in the BO server farm.  At least with V1.05, we can decode and display with the new stations file format -- just have to wait to see when the rest of the statistics are fixed.
Ken True/Saratoga, CA, USA main site: saratoga-weather.org
Davis VP1+ FARS, Boltek-PCI/NexStorm, microSferics ToA, Blitzortung RED, GRLevel3, WD, WL, VWS, Cumulus, Meteobridge/hub
Free weather PHP scripts/website templates - update notifications on Twitter saratogaWXPHP

Offline PaulMy

  • Forecaster
  • *****
  • Posts: 4134
    • KomokaWeather
Re: Blitzortung back, but stations file structure changed
« Reply #12 on: October 31, 2018, 11:14:22 AM »
Thanks guys...  I have been capturing a screen shot of my station every day and haven't been able for a few days, and presumed it was the server change.  No worries.

Enjoy,
Paul
Davis Vantage Pro 2 Plus 24-FARS Wireless
Envoy/USB DataLogger/Cumulus v.1.9.4   -   Vue Console/WiFiLogger
Komoka, ON  Canada
www.komokaweather.ca   www.komokaweather.com   www.komokaweather.com/weather28/   www.komokaweather.com/pws/index.php
Blitzortung Station #1076 www.lightningmaps.org/blitzortung7/america/index.php?bo_page=statistics&bo_show=network&lang=en&bo_station_id=969

Offline W0BTU

  • Member
  • *
  • Posts: 19
Re: Blitzortung back, but stations file structure changed
« Reply #13 on: October 31, 2018, 01:02:41 PM »
I am still on firmware 8.0-b2 should I update?

Hello Paul,
Although upgrading the firmware isn't going to fix these issues, 9.1 is far superior. The best feature is how much better it discriminates between bona fide strikes and noise.

Regards, Mike
Stations: 1977, 2294

Offline vk2him

  • Member
  • *
  • Posts: 4
Re: Blitzortung back, but stations file structure changed
« Reply #14 on: November 01, 2018, 06:36:50 PM »
The stations file seems to have 'stuck' statistics for stations .. only the timestamp seems to change.  I think it's an update issue in the BO server farm.  At least with V1.05, we can decode and display with the new stations file format -- just have to wait to see when the rest of the statistics are fixed.

Hi Ken, any chance you may be able to send me the BOMaps scripts please? I've been using the mybo templates till the recent changes stopped it working.  Here's a link to my stats:
https://www.lightningmaps.org/blitzortung/oceania/index.php?bo_page=statistics&bo_show=station&bo_sid=2039&bo_page=statistics&bo_show=station&bo_sid=2039

Thanks in advance
Ian