Topic: Server Problems?

[Knorrli]

Since about 24 h my Blitzortung maps don't display strokes and tracking stations anymore; date/time info in top right corner is accurate. Is the problem on my side or is the server partially down for testing?

[saratogaWX]

The last several days, the data.blitzortung.de server has been having really slow response at times.  A query for .json file that normally takes < 10 seconds takes upwards of 100-200 seconds.  I've had to adjust my bash script to run the BOmaps animations to kill off any lingering get-BO-maps.php processes that haven't finished in 5 minutes

Hope they fix the issue soon.. sure is a pain.

[miraculon]

I don't see strike updates on Blitzortung.org, but Lightningmaps.org seems OK.

The data feeds for my scripts are also OK. As Ken indicated, there have been intermittent problems with this data and my maps we going "stale" yesterday.

Greg H.

[Knorrli]

Thanks guys, I can live with "stale" for a while, at least I don't have to start taking my computer apart 

[miraculon]

If I can quote myself,

I don't see strike updates on Blitzortung.org

Actually, they are updating on the 2nd tab "Live Overview".

Greg H.

[W3DRM]

The last several days, the data.blitzortung.de server has been having really slow response at times.  A query for .json file that normally takes < 10 seconds takes upwards of 100-200 seconds.  I've had to adjust my bash script to run the BOmaps animations to kill off any lingering get-BO-maps.php processes that haven't finished in 5 minutes

Hope they fix the issue soon.. sure is a pain.

Ken,

I just noticed that my last update for BOmaps was July 1st. My status.txt file shows the following:

• BOmapgen - V1.01 - 21-Apr-2015
  start processing for region 3 data Mon, 06 Jul 2015 18:45:01
  Fatal error:  Maximum execution time of 30 seconds exceeded in /homepages/14/d239618093/htdocs/BOmaps/BOmapgen-inc.php on line 84

How did you modify you bash script to kill off the lingering processes?

[saratogaWX]

I changed the animate.txt file from

Code: [Select]

cd $HDIR
$PHPcmd -q gen-BO-maps.php > status.txt 2>&1
to

Code: [Select]

cd $HDIR
ps auxw | grep "gen-BO-maps.php" | grep -v 'grep' | awk '{print $2}' | xargs kill >/dev/null 2>&1
$PHPcmd -q gen-BO-maps.php > status.txt 2>&1


[W3DRM]

I changed the animate.txt file from

Code: [Select]

cd $HDIR
$PHPcmd -q gen-BO-maps.php > status.txt 2>&1
to

Code: [Select]

cd $HDIR
ps auxw | grep "gen-BO-maps.php" | grep -v 'grep' | awk '{print $2}' | xargs kill >/dev/null 2>&1
$PHPcmd -q gen-BO-maps.php > status.txt 2>&1


Thanks Ken. Do I have to stop and then restart the cron job to get it to use the newly modified animate.txt file?

[saratogaWX]

If the cron job just executes the animate.txt script, then just update the contents of animate.txt and FTP it to your site.
Remember to use Notepad++ to edit the file, and show symbols end-of-line to make sure only New-Line characters (\n) appear.
If the lines end in CR,LF  (\r\n), the bash shell will barf and not execute the script.

[W3DRM]

If the cron job just executes the animate.txt script, then just update the contents of animate.txt and FTP it to your site.
Remember to use Notepad++ to edit the file, and show symbols end-of-line to make sure only New-Line characters (\n) appear.
If the lines end in CR,LF  (\r\n), the bash shell will barf and not execute the script.

I don't see the \n or CR,LF at the end of the lines but, I do see LF on all lines.

Am still getting the same error as before. I notice the strikes.txt file in the cache folder is 684,677,427 bytes in size. Is that reasonable? It has a timestamp that matches the last run time of the cron job.

[saratogaWX]

LF is just fine (Line-feed or new-line).

600+MB is way too large for the strikes.txt file.  Upload a 0 byte strikes.txt file to clear it out, then the collection will restart from the last two hours.

My strikes.txt is at 489.34 KB right now.

[miraculon]

Don,

Log onto sferics.us and see this: http://sferics.us/bo1/index.php?topic=141.msg679#msg679

It sounds like you didn't pick up v1.03.

I saw the update while I was out of town.

Can I just upload the newest version of BOmapgen-inc.php and BOstations-inc.php and call it good?

I would assume that this should be OK and that I don't need to upload the entire new archive...

UPDATE: I went ahead and did upload just these two files from V1.03 and it seems OK so far...

Greg H.

FWIW, my file size is

242.99 KB is strikes.txt file size used by Blitzortung.

I uploaded the two files that were part of the v1.03 update.

Greg H.

[W3DRM]

Wow, how did I miss the update to V1.03? Anyway, I have created a new strikes.txt file and downloaded the V1.03.

Will now watch to see what happens. Hopefully, this will cure my BOmap problems.

Thanks for the help guys!! 

EDIT: After a couple minutes I checked the status.txt file. Here is what I now see. Looks like it is now rebuilding the history files.

BOmapgen - V1.03 - 22-May-2015
Parms: URLPath='/BOmaps/' for map gen
Parms: maxFilesize='200 MB' for cache/strikes.txt
Parms: ourTZ='America/Los_Angeles'
start processing for region 3 data Tue, 07 Jul 2015 14:50:01
current cache/strikes.txt strikes file size is 0 B
cache/strikes.txt filtered for old data
First data: Wed, 31 Dec 1969 16:00:00
Last data : Tue, 07 Jul 2015 12:50:01
old data - restart collection from Tue, 07 Jul 2015 12:50:01
fetching new strikes file at http://user:pass-omitted@data.blitzortung.org/Data_3/Protected/Strikes/2015/07/07/19/50.json

[Knorrli]

The Live Maps still doesn't track strikes and the Live Overview Map seems to be very sluggish. The Live Map shows a total of over 89,000 lightning. Is this the problem?

[W3DRM]

I don't understand what is going on. I still don't see any updates to my BO-SWN lightning pages. Looking at the status.txt seems to show a "HTTP/1.1 404 Not Found" error but i don't know what that means in this situation. The cron job seems to be running okay.

• http://www.carsonvalleyweather.com/BOmaps/status.txt

[saratogaWX]

There was another very slow period for data.blitzortung.de JSON requests.. one cycle for me took 5000 seconds to complete.

I had to stop cron, run gen-BO-maps.php manually (via SSH/shell), then restart the cron.  Even now it can take 80 seconds to complete with just one fetch.

[W3DRM]

There was another very slow period for data.blitzortung.de JSON requests.. one cycle for me took 5000 seconds to complete.

I had to stop cron, run gen-BO-maps.php manually (via SSH/shell), then restart the cron.  Even now it can take 80 seconds to complete with just one fetch.

What is confusing me is that when I bring up the individual BOSWN-ani.gif file I see the old July 1 images. The modified date shown via FileZilla shows 7/7/2015 at a current time. To me, it appears the cron file, while running, isn't grabbing and/or processing the latest data. Is this a symptom of the server being so slow? This cron stuff is giving me a headache. 

Perhaps I should do the same thing you did and stop the cron job and run gen-BO-maps.php manually and see what happens.

[saratogaWX]

Yes, the slow responses from data.blitzortung.de are what is causing the issues.  I do hope they fix that soon!

Yes, stop your cron running of animate.txt, kill any gen-BO-maps.php process(s) and run gen-BO-maps.php and watch how slow it goes.  When it finishes, then engage the cron again and it may be able to complete with 1 or 2 fetches to freshen the file even if it takes 120 seconds to fetch one.. as long as it completes in < 300 seconds, the next cron invocation won't kill the long-running one.

[W3DRM]

Thanks Ken,

Well, just as I was getting ready to kill the cron job, I took a quick look at the BOSWN and other images and discovered that all of them are now showing current lightning data. So, I guess things have worked themselves out for the time being. I'll continue to watch for a while to see if the problem starts all over again.

Have you made any contact with Tobi or Egon about the server issues? If not, I'll send him a note and see what they say.

[saratogaWX]

I've not contacted Tobi or Egon about the slow response.. please contact them!

I'd love to have the queries to their server take 10 secs or less for each 10-minute slice.

[miraculon]

Yes, the slow responses from data.blitzortung.de are what is causing the issues.

Ken, I am a little puzzled by this. The scripts point to data.blitzortung.org. If I ping each, I get differing IP address responses.

Are they mirrored servers?

Greg H.

[saratogaWX]

I get it as one server:

81.7.10.149

[miraculon]

data.blitzortung.de resolves as 81.7.10.149 for me and data.blitzortung.org resolves as 217.145.98.148.
What nameserver are you using? I have 8.8.8.8 (Google) set in my router.

Code: [Select]

Pinging data.blitzortung.de [81.7.10.149] with 32 bytes of data:
Reply from 81.7.10.149: bytes=32 time=142ms TTL=47
Reply from 81.7.10.149: bytes=32 time=139ms TTL=47
Reply from 81.7.10.149: bytes=32 time=161ms TTL=47
Reply from 81.7.10.149: bytes=32 time=142ms TTL=47

Ping statistics for 81.7.10.149:
    Packets: Sent = 4, Received = 4, Lost = 0 (0% loss),
Approximate round trip times in milli-seconds:
    Minimum = 139ms, Maximum = 161ms, Average = 146ms

C:\Users\Greg>ping data.blitzortung.org

Pinging data.blitzortung.org [217.145.98.148] with 32 bytes of data:
Reply from 217.145.98.148: bytes=32 time=130ms TTL=43
Reply from 217.145.98.148: bytes=32 time=132ms TTL=43
Reply from 217.145.98.148: bytes=32 time=133ms TTL=43
Reply from 217.145.98.148: bytes=32 time=133ms TTL=43

Ping statistics for 217.145.98.148:
    Packets: Sent = 4, Received = 4, Lost = 0 (0% loss),
Approximate round trip times in milli-seconds:
    Minimum = 130ms, Maximum = 133ms, Average = 132ms
I tried several DNS addresses on http://www.kloth.net/services/nslookup.php that I found on http://public-dns.tk/nameserver/us.html and they seem to be as shown above.

Greg H.