Author Topic: USPLN Google Earth - Extracting the Stroke Text Data  (Read 510 times)

0 Members and 1 Guest are viewing this topic.

Offline waysta

  • Contributor
  • ***
  • Posts: 131
USPLN Google Earth - Extracting the Stroke Text Data
« on: April 27, 2017, 08:51:18 PM »
I've been searching for a way to record my local strike data without having to write it down from the screen list; I think this works.  I am a total novice with Google Earth (GE), so others may know of an easier way.

On GE, left panel, click through the arrows to open up "Lightning Data", "Detailed View", then "Strikes in View" (I think that is your current screen area).

Then, right click on "Strikes in View" and email the .KMZ file to yourself.  After you receive the .KMZ file, rename the extension to .zip and unzip the file.  You get a .KML file that contains all of your strikes in text.  It's still embedded in lots of extraneous code, but at least the recently recorded strike data is all there in readable form.  I suppose a small amount of code could put just the entries into a list or excel, or something like that.  Maybe there are canned programs to work with extracting data from .KML files?

Anyway, the individual strike data there (date, time, polarity, lat, lon, current)!  Now, I can think about my amateur science experiments using the data.
« Last Edit: April 27, 2017, 09:06:13 PM by waysta »

Offline Bushman

  • Forecaster
  • *****
  • Posts: 6444
    • Eagle Bay Weather

Offline miraculon

  • Sunrise Side Weather
  • Forecaster
  • *****
  • Posts: 2769
  • KE8DAF
    • Sunrise Side Weather in Rogers City MI USA
Re: USPLN Google Earth - Extracting the Stroke Text Data
« Reply #2 on: April 28, 2017, 08:27:22 AM »
Thanks to both of you. It works pretty slick!  \:D/

Greg H.




Blitzortung Stations 668, 706 - CWOP CW4114 -  CoCoRaHS MI-PI-1

Offline DaleReid

  • Forecaster
  • *****
  • Posts: 1111
    • Weather at Eau Claire, WI
Re: USPLN Google Earth - Extracting the Stroke Text Data
« Reply #3 on: April 28, 2017, 09:40:59 AM »
My station is in 'review' mode right now for the last few days, and Sandy hasn't sent me my access code yet, so I'm not sure of all the things you guys are seeing.

May I ask what you're doing with these data when extracted?  Maybe it will be obvious when I get my access, but does this give range, direction, estimated amperage, etc?

ECWx.info
&
ECWx.info/wx/index.php

Offline Bushman

  • Forecaster
  • *****
  • Posts: 6444
    • Eagle Bay Weather
Re: USPLN Google Earth - Extracting the Stroke Text Data
« Reply #4 on: April 28, 2017, 09:54:03 AM »
I am waiting as well, but pages 14-16 here seem to describe the data captured:  http://www.uspln.com/documentation/Google_Earth_Lightning_Service_Feb2016.pdf

Offline DaleReid

  • Forecaster
  • *****
  • Posts: 1111
    • Weather at Eau Claire, WI
Re: USPLN Google Earth - Extracting the Stroke Text Data
« Reply #5 on: April 28, 2017, 10:08:11 AM »
Bushman:
That document is a real help in knowing what to do and how to play around getting used to the system.

I'll study it a bit more over a cup of coffee and put my feet up to read it later today.

Thanks for sharing it.  The data we'll be able to view is worth the effort in getting the station up and running for them.  I just hope that they'll only need a few more days of monitoring before they certify it for use. 

Thanks again, sort of like waiting for Christmas!  Dale
ECWx.info
&
ECWx.info/wx/index.php

Offline Bushman

  • Forecaster
  • *****
  • Posts: 6444
    • Eagle Bay Weather
Re: USPLN Google Earth - Extracting the Stroke Text Data
« Reply #6 on: April 28, 2017, 10:13:23 AM »
If GE stores the lightning KML the way the document suggests, it is pretty easy to traverse the file system and extract the data, possibly without using a conversion to Excel.  Once my system gets up I will take a look at it and see if a bit of code can help USLPN users.

Offline waysta

  • Contributor
  • ***
  • Posts: 131
Re: USPLN Google Earth - Extracting the Stroke Text Data
« Reply #7 on: April 28, 2017, 10:34:10 AM »
May I ask what you're doing with these data when extracted?  Maybe it will be obvious when I get my access, but does this give range, direction, estimated amperage, etc?

yep, Date, Time, Lat, Lon, type (cloud,neg,pos), est current for ground strokes

I might look at gamma rays or X-rays on occasion (X-ray emissions, and there has been discussion of cosmic ray lightning triggers), not sure. 

For most types of amateur science experiments, the data (see the graphics of "bubbles" on page 13 of the pub cited above), text or ASCII data is needed (or, you have to do a LOT of OCR, typing, or writing by pen or pencil).  I was looking for the local limited data version of what universities get free access to: http://www.unidata.ucar.edu/data/lightning/napln.html Probably way too much volume 24/7, (NAPLN data are made available to universities at no cost over the Internet for educational and research purposes)  I was told only for Universities, not available to amateur scientists, host or not (possibly a blessing in disguise, because at least for now, I need relatively little data compare to a 24/7 real time data stream).
« Last Edit: April 28, 2017, 10:58:34 AM by waysta »

Offline waysta

  • Contributor
  • ***
  • Posts: 131
Re: USPLN Google Earth - Extracting the Stroke Text Data
« Reply #8 on: April 28, 2017, 10:44:34 AM »
If GE stores the lightning KML the way the document suggests, it is pretty easy to traverse the file system and extract the data, possibly without using a conversion to Excel.  Once my system gets up I will take a look at it and see if a bit of code can help USLPN users.

That would be great, do you think the data file is on our own machines?  Or, does the Network link server feed a GE server, and we see the resultant GE page from a GE server somewhere?

The KML file provided does not include the stroke data, only the network URL to the GE USPLN Network link server.

So, when I get the KMZ emailed file by right clicking on the menu list to the left of the page, is that coming from a GE server (I think might be the case), or from a file on my local hard drive?
« Last Edit: April 28, 2017, 10:53:12 AM by waysta »

Offline Bushman

  • Forecaster
  • *****
  • Posts: 6444
    • Eagle Bay Weather
Re: USPLN Google Earth - Extracting the Stroke Text Data
« Reply #9 on: April 28, 2017, 10:58:22 AM »
Hmmm... probably not local based on your description.  But a GET (or WGET) might still be able to retrieve the data.  More research needed.

Offline WeatherHost

  • Forecaster
  • *****
  • Posts: 2694
Re: USPLN Google Earth - Extracting the Stroke Text Data
« Reply #10 on: April 28, 2017, 11:08:50 AM »
Whatever you do, remember the agreements signed and don't get the program cancelled.

Banners, tags, widgets, etc. are why I have Signatures turned off.

Offline Bushman

  • Forecaster
  • *****
  • Posts: 6444
    • Eagle Bay Weather
Re: USPLN Google Earth - Extracting the Stroke Text Data
« Reply #11 on: April 28, 2017, 11:33:37 AM »
Whatever you do, remember the agreements signed and don't get the program cancelled.

Good reminder!  Personal use ONLY!!

Offline waysta

  • Contributor
  • ***
  • Posts: 131
Re: USPLN Google Earth - Extracting the Stroke Text Data
« Reply #12 on: April 28, 2017, 12:53:02 PM »
Whatever you do, remember the agreements signed and don't get the program cancelled.

I think a similar caution goes to direct reads from the server.  It looks like we are set up for 30 second updates.  Were someone to make near real-time reads, we might exceed our access rights/permissions.

To my best understanding, emailing my own authorized data access (e.g. a GE screen of data) using a standard available GE feature to myself is well within our agreement for personal use.  I told them about the email method too.  I should be so lucky that others would want the results of my amateur science experiments.

If I ever do any amateur science worthy of academic publication, I would simply review the academic rules, and ask for permission.

My impression is that we are providing a valuable service to them by hosting stations, and within some limits we are free to perform our own amateur weather observations and amateur science experiments.  Was that not the whole point, we provide labor for installation at no cost, as well as leased space for their antenna and sensor package at no cost, including power, internet, local monitoring, and maintenance, to a for profit commercial enterprise (TOA, The Weather Company, and IBM), in exchange for data (that except for universities is otherwise sold at cost) for our own hobby use.

There might be some question even if we want to post data here amongst ourselves, so we can just ask them if it is okay, should that situation arise.  (note that a "private area" is not a solution, a violation of an agreement is a violation whether in public or private.  Similarly, the agreement is not in any way between any of us.  Each agreement is between each one of us individually and the company.)
« Last Edit: April 28, 2017, 02:01:34 PM by waysta »

Offline waysta

  • Contributor
  • ***
  • Posts: 131
Re: USPLN Google Earth - Extracting the Stroke Text Data
« Reply #13 on: April 28, 2017, 01:02:46 PM »
One possibly marginal use that comes to mind, is comparing this data to Blitzortung data.  Surely for personal use, it would be okay.  However, what about giving feedback or posting differences, possibly not?

Fortunately, a non-issue for me anyway, because Blitzortung looks to be unobtanium for this season.
« Last Edit: April 28, 2017, 01:41:13 PM by waysta »

Offline WeatherHost

  • Forecaster
  • *****
  • Posts: 2694
Re: USPLN Google Earth - Extracting the Stroke Text Data
« Reply #14 on: April 29, 2017, 04:52:39 AM »
I gave it a shot, but it comes in a .kmz file which the converter doesn't handle.  If I unzip it, I can open it in Notepad and see some data.

But it appears that it's all strikes in view based on how you have the map zoomed and not just those for your sensor.  That doesn't really do me any good.  I want to see the ones my sensor has detected.  That may require the USB bit.

Banners, tags, widgets, etc. are why I have Signatures turned off.

Offline waysta

  • Contributor
  • ***
  • Posts: 131
Re: USPLN Google Earth - Extracting the Stroke Text Data
« Reply #15 on: April 30, 2017, 10:27:23 AM »
I gave it a shot, but it comes in a .kmz file which the converter doesn't handle.  If I unzip it, I can open it in Notepad and see some data.

But it appears that it's all strikes in view based on how you have the map zoomed and not just those for your sensor.  That doesn't really do me any good.  I want to see the ones my sensor has detected.  That may require the USB bit.

skipped a step: "After you receive the .KMZ file, rename the extension to .zip and unzip the file.  You get a .KML file that contains all of your strikes in text. "  It is the KML file that you convert.

If you want more data - zoom out.

You are right about what our individual sensor detected.  There probably is a lightning map lines equivalent somewhere, but not one that we get to see.  USB "strokes report" only advances the stroke count, there is no specific single stroke data there, that would be nice.

It might be possible to mirror the ethernet port with an ethernet switch (or a router with port mirror capability) to look at the individual reports, however they probably need to be decoded, not sure how difficult that would be with something like WireShark.

On the other hand, the time of the report based on local computer time could work, if the detection rate is low enough to just watch all of them simultaneously.
« Last Edit: April 30, 2017, 10:57:25 AM by waysta »

Offline WeatherHost

  • Forecaster
  • *****
  • Posts: 2694
Re: USPLN Google Earth - Extracting the Stroke Text Data
« Reply #16 on: April 30, 2017, 12:29:10 PM »
No, I didn't miss that step, but I think what I did was try to convert the wrong file.  The output from unzipping ends up being doc.kml instead of strikesinview.kml


At any rate, it's of no value to me personally unless I can limit to my sensor which doesn't appear possible.

Banners, tags, widgets, etc. are why I have Signatures turned off.

Offline Bushman

  • Forecaster
  • *****
  • Posts: 6444
    • Eagle Bay Weather
Re: USPLN Google Earth - Extracting the Stroke Text Data
« Reply #17 on: April 30, 2017, 12:41:24 PM »
There has to be a geographic coordinate system buried in the stream somewhere otherwise you could not calculate distances, etc.

Offline waysta

  • Contributor
  • ***
  • Posts: 131
Re: USPLN Google Earth - Extracting the Stroke Text Data
« Reply #18 on: April 30, 2017, 12:47:35 PM »
There has to be a geographic coordinate system buried in the stream somewhere otherwise you could not calculate distances, etc.

The stream might just include our station i.d. (the central computer probably knows each of our station locations), or, it could include our lat, lon each time too.  It's a pretty short transmission.

>>At any rate, it's of no value to me personally unless I can limit to my sensor which doesn't appear possible.

  I tried one method which is far less than ideal: Set up your USB monitoring if you have not already done that.  Everytime you hit return where a previous command was "strokes", you see the updated count for your station.

  On a not so crazy high stroke rate day, watch the lightning map (Blitz), while periodically hitting return to see when your stroke count advances.  Remember, we don't have the long distance operation that Blitz has with its multiple magnetic solenoid antennas.  Even though a very crude comparison, it was pretty easy to see some of the correlations.
« Last Edit: April 30, 2017, 02:47:00 PM by waysta »

Offline DaleReid

  • Forecaster
  • *****
  • Posts: 1111
    • Weather at Eau Claire, WI
Re: USPLN Google Earth - Extracting the Stroke Text Data
« Reply #19 on: April 30, 2017, 12:54:42 PM »
I'd wager the location data (lat, long, elevation) is encoded with each data transmission, for safety sake. Or one can retransmit every 1/2 hour or so, as the gps coordinates tend to wander a bit.

With the non-burden of the size of transmission (unlike the deep space network with their baud rate listening to Cassini for instance) I'd go for encoding the coordinates at the time of capture.  Or maybe the box has smarts enough like the Blitz does to do a smoothing over time, which of course is best of all.  I remember when a surveyor friend of mine had a new $80,000 (at that time) gizmo that he had in his truck all the time and he stopped by my place, drove a nail and let it run for six or eight hours and got the best reading he could for me at the time. 

All interesting thoughts on what the best way is.  I'm thinking there are a lot more out there and hope that others give their ideas too since I'm operating on old info and faulty memory. 

Dale
ECWx.info
&
ECWx.info/wx/index.php

Offline Bushman

  • Forecaster
  • *****
  • Posts: 6444
    • Eagle Bay Weather
Re: USPLN Google Earth - Extracting the Stroke Text Data
« Reply #20 on: April 30, 2017, 02:33:48 PM »
It would be interesting to poke a bit at the data stream...

Offline waysta

  • Contributor
  • ***
  • Posts: 131
Re: USPLN Google Earth - Extracting the Stroke Text Data
« Reply #21 on: April 30, 2017, 07:21:44 PM »
I'd wager the location data (lat, long, elevation) is encoded with each data transmission, for safety sake. Or one can retransmit every 1/2 hour or so, as the gps coordinates tend to wander a bit.

... Or maybe the box has smarts enough like the Blitz does to do a smoothing over time, which of course is best of all.  ...

They might average our gps location reporting at central during the few days to week commissioning time too ...

 

anything