Topic: Weather Template and scripts from Leuven/Belgium/Europe

[saratogaWX]

Over time, websites that we use as source for our own sites will change (and maybe become unusable).  Most recently, the NWS decision to shutdown weather.noaa.gov (our source for METAR info, amongst other raw data) in favor of tgftp.nws.noaa.gov site.

Wunderground has been morphing too - the old 'classic' site is long gone, and they're really focusing on apps and main-site mobile friendliness.  I wouldn't be supprised if they switch to HTTPS in the near future, and that would cause some of our scripts to be redone to handle that.  Such is the life of 'data scrapers'.. I've been at it for 10 years now, and see the future as more of the same adapting to what the source sites do (generally without much prior notice).

As a template developer, I really appreciate it when Wim or Jachym answer questions regarding my scripts/template -- their advice is good, and being in a different timezone, extends the coverage hours for responses.  So, thanks to both Wim and Jachym for doing that!  I also appreciate the forum members who have 'been there, fixed that' when they answer questions based on experience with the problem -- it saves me from having to respond to issues that have happened before/been resolved before.  That's what the forums are about -- members helping members and all of us learning and advancing our mutual weather hobby.

Best regards,
Ken

[Jáchym]

I agree with you Ken, I personally see HTML scraping as the very last resort, it is always problematic and always means sooner or later you as a developer will have problems with it. And more and more often major sites try to disable this because obviously they do not want others to use their server resources and duplicate content. Fortunately, more and more sites also offer their own API, which is much safer to use and also much less vulnerable to any future problems. Its just that obviously you can only use what they want you to be able to use, but then the other thing is that thats also the only thing you should be using anyway, if they dont want certain parts of the page to be used by other websites (eg. the WU site above), then trying to find a workaround or scraping it would 99% be against the terms anyway.

[saratogaWX]

I agree that page-scraping is the last resort, and that using a provided API is preferable.  Some sites API is inadequate (the EC for forecasts for example), so a page-scrape is needed to provide both icons and text for a forecast there.  My scripts have been converted from scrapers to API-users where the APIs exist (Wunderground forecast, USGS earthquakes, NWS alerts, USNO sun/moon) and will continue to be converted as new APIs appear.

[txweather.org]

Thank you all for everything. Ken, Wim, Jachym, Curly, The whole crue!
You guys are awesome and what you guys are doing for the community its priceless.

Thank you all again and to me all of you guys opinions are priceless.

[stewart45]

somehow I have screwed up my website main page , since moving over to Meteobridge, it appeared to work for a while , the ajax dashboard has not been updating now for a couple of days, correct data is being uploaded every 2mins to the right area uploadMB not sure where to look now

[Jáchym]

I think the problem is in your mbTags.
Looking at your webpage and the console, the Ajax calls work fine, but the results fed to the page show the zeros.

In fact if you look in the mbTags.txt you attached, you will see that for example the T-act, which in MB is the current temperature, is 0.

[wvdkuil]

somehow I have screwed up my website main page , since moving over to Meteobridge, it appeared to work for a while , the ajax dashboard has not been updating now for a couple of days, correct data is being uploaded every 2mins to the right area uploadMB not sure where to look now

As @Jáchym already pointed to, your upload files uploadMB/tagsMB.txt contains  at random times  only zeroes or -- values.  But most of the times the values are quite normal. If this happen again, check the live data page, maybe the communication between MB and the console failed.
You can check here:  http://www.skeltonweather.uk/weather2/index.php?compare&p=0000
Normally the tagsMB.txt is uploaded every 5 minutes.
With a faster frequency the realtime is uploaded also. http://www.skeltonweather.uk/weather2/uploadMB/realtime.txt  Newer values from the realtime files are merged with the 5 minutes values.   Those combined values should be displayed on your startpage.

As far as I can see the values seems correct at this time, example here: http://www.skeltonweather.uk/weather2/index.php?p=historyv3

The only error I see is that you did not yet update the links to get correct METAR data.  So the current conditions are not there and the icon is the "question-mark" one.  There is a big warning EXTREMELY IMPORTANT and a link to a how-to document at all the support pages,  leading to this pdf http://leuven-template.eu/noaa.pdf

Wim

[Jáchym]

somehow I have screwed up my website main page , since moving over to Meteobridge, it appeared to work for a while , the ajax dashboard has not been updating now for a couple of days, correct data is being uploaded every 2mins to the right area uploadMB not sure where to look now

As @Jáchym already pointed to, your upload files uploadMB/tagsMB.txt contains  at random times  only zeroes or -- values.  But most of the times the values are quite normal. If this happen again, check the live data page, maybe the communication between MB and the console failed.
You can check here:  http://www.skeltonweather.uk/weather2/index.php?compare&p=0000
Normally the tagsMB.txt is uploaded every 5 minutes.
With a faster frequency the realtime is uploaded also. http://www.skeltonweather.uk/weather2/uploadMB/realtime.txt  Newer values from the realtime files are merged with the 5 minutes values.   Those combined values should be displayed on your startpage.

As far as I can see the values seems correct at this time, example here: http://www.skeltonweather.uk/weather2/index.php?p=historyv3

The only error I see is that you did not yet update the links to get correct METAR data.  So the current conditions are not there and the icon is the "question-mark" one.  There is a big warning EXTREMELY IMPORTANT and a link to a how-to document at all the support pages,  leading to this pdf http://leuven-template.eu/noaa.pdf

Wim

Thanks Wim for more detailed analysis, I see it now working too, but when I looked about an hour and half ago, shortly after he posted it, all the current values were 0 and that is what the page showed.

So this clearly shows that the scripts and their setup are fine. Otherwise it would never work. It must be a problem in the communication between MB and the PWS.

[stewart45]

Thanks everyone, yes it appeared to be a comms problem, also the Lat setting in the meteobridge software had a strange corruption, where there should have been the Latitude reading for my site had been substituted by the word "meteobridge" very strange anyway seems to be sorted now just have to sort out the current weather icon, thanks again,
Jachym, like your Olympic's banner

[Jáchym]

LOL, thanks

and with regards to your page I guess neither me nor Wim can help you, you must find the source of the comms problem and if unsure contact Boris

[Météo Québec]

  On my Leuven website the last rained show 31-12 @ 19:00 since yesterday morning... And its not the first time it happen. I look in my WD webfiles  the testtags and tagsWD and both show my last rained correctly. the tag of wsAjaxDashboard_v3.php is set up like this:    
<p><span class="lTxt"><?php echo langtransstr("Last rain on").":";?></span><span class="rTxt"><?php echo $ws['lastRainTip'].string_date($ws['lastRained'],$SITE['dateMDFormat'].' @ '.$SITE['timeOnlyFormat']); ?></span></p>

Also the nearby metar is not uploading anymore, i have check in the ca_multi_fct.txt and all the metar codes are correct.

For the burtime (Sunburn in), does someone know what is the proper tag?

Last thing, in the UV section the UV 8 (red) should be in the very high section (red) and not in the high section (orange).

Thanks

Steven

[wvdkuil]

On my Leuven website the last rained show 31-12 @ 19:00 since yesterday morning... And its not the first time it happen. I look in my WD webfiles  the testtags and tagsWD and both show my last rained correctly. the tag of wsAjaxDashboard_v3.php is set up like this:    
<p><span class="lTxt"><?php echo langtransstr("Last rain on").":";?></span><span class="rTxt"><?php echo $ws['lastRainTip'].string_date($ws['lastRained'],$SITE['dateMDFormat'].' @ '.$SITE['timeOnlyFormat']); ?></span></p>

Also the nearby metar is not uploading anymore, i have check in the ca_multi_fct.txt and all the metar codes are correct.

For the burtime (Sunburn in), does someone know what is the proper tag?

Last thing, in the UV section the UV 8 (red) should be in the very high section (red) and not in the high section (orange).

Thanks

Steven

sorry, I am abroad without proper internet. Will look into this after 28 August. Wim

[Météo Québec]

On my Leuven website the last rained show 31-12 @ 19:00 since yesterday morning... And its not the first time it happen. I look in my WD webfiles  the testtags and tagsWD and both show my last rained correctly. the tag of wsAjaxDashboard_v3.php is set up like this:    
<p><span class="lTxt"><?php echo langtransstr("Last rain on").":";?></span><span class="rTxt"><?php echo $ws['lastRainTip'].string_date($ws['lastRained'],$SITE['dateMDFormat'].' @ '.$SITE['timeOnlyFormat']); ?></span></p>

Also the nearby metar is not uploading anymore, i have check in the ca_multi_fct.txt and all the metar codes are correct.

For the burtime (Sunburn in), does someone know what is the proper tag?

Last thing, in the UV section the UV 8 (red) should be in the very high section (red) and not in the high section (orange).

Thanks

Steven

sorry, I am abroad without proper internet. Will look into this after 28 August. Wim

No problem Wim, thanks.

[djroby19]

Hello Wim wanted to warn you that the cloud radar invisible how can I fix ?
Thank you for the work you do for us
www.meteopomezia.altervista.org
Thanks 

[wvdkuil]

Hello Wim wanted to warn you that the cloud radar invisible how can I fix ?
Thank you for the work you do for us
www.meteopomezia.altervista.org
Thanks 

i am back after August 28 and I can check what is causing this, sorry for the delay
Wim

[djroby19]

No problem Wim....

[Jáchym]

It is not a problem with the template, the entire sat24 page is currently inaccessible.

http://www.sat24.com/

[wvdkuil]

On my Leuven website the last rained show 31-12 @ 19:00 since yesterday morning... And its not the first time it happen. I look in my WD webfiles  the testtags and tagsWD and both show my last rained correctly. the tag of wsAjaxDashboard_v3.php is set up like this:    
<p><span class="lTxt"><?php echo langtransstr("Last rain on").":";?></span><span class="rTxt"><?php echo $ws['lastRainTip'].string_date($ws['lastRained'],$SITE['dateMDFormat'].' @ '.$SITE['timeOnlyFormat']); ?></span></p>

I am not familiar with that code or the function used. Please ask the person who amended your script for advice.

Also the nearby metar is not uploading anymore, i have check in the ca_multi_fct.txt and all the metar codes are correct.

The weather.noaa.gov website officially stopped working in June this year, although it seemed to continue for a few months extra.  There are multiple discussions on the fora. Check the Leuven-Template support site: http://support.leuven-template.eu/  There is a VERY LARGE message on each page which links to => a pdf at: http://leuven-template.eu/noaa.pdf
You should modify this line 46 in metar/wsMetarTxt.php

Code: [Select]

$metarURL = 'http://weather.noaa.gov/pub/data/observations/metar/stations/'. $icao.'.TXT';to

Code: [Select]

$metarURL = 'http://tgftp.nws.noaa.gov/data/observations/metar/stations/'. $icao.'.TXT';

For the burtime (Sunburn in), does someone know what is the proper tag?

Last thing, in the UV section the UV 8 (red) should be in the very high section (red) and not in the high section (orange).

There are different descriptions for that.  I will put a setting in the next release of the script to choose the range for each color and to choose which texts should be used.

Thanks
Steven

Succes, wim

[Météo Québec]

On my Leuven website the last rained show 31-12 @ 19:00 since yesterday morning... And its not the first time it happen. I look in my WD webfiles  the testtags and tagsWD and both show my last rained correctly. the tag of wsAjaxDashboard_v3.php is set up like this:    
<p><span class="lTxt"><?php echo langtransstr("Last rain on").":";?></span><span class="rTxt"><?php echo $ws['lastRainTip'].string_date($ws['lastRained'],$SITE['dateMDFormat'].' @ '.$SITE['timeOnlyFormat']); ?></span></p>

I am not familiar with that code or the function used. Please ask the person who amended your script for advice.

Also the nearby metar is not uploading anymore, i have check in the ca_multi_fct.txt and all the metar codes are correct.

The weather.noaa.gov website officially stopped working in June this year, although it seemed to continue for a few months extra.  There are multiple discussions on the fora. Check the Leuven-Template support site: http://support.leuven-template.eu/  There is a VERY LARGE message on each page which links to => a pdf at: http://leuven-template.eu/noaa.pdf
You should modify this line 46 in metar/wsMetarTxt.php

Code: [Select]

$metarURL = 'http://weather.noaa.gov/pub/data/observations/metar/stations/'. $icao.'.TXT';to

Code: [Select]

$metarURL = 'http://tgftp.nws.noaa.gov/data/observations/metar/stations/'. $icao.'.TXT';

For the burtime (Sunburn in), does someone know what is the proper tag?

Last thing, in the UV section the UV 8 (red) should be in the very high section (red) and not in the high section (orange).

There are different descriptions for that.  I will put a setting in the next release of the script to choose the range for each color and to choose which texts should be used.

Thanks
Steven

Succes, wim

Thanks for your respond Wim The nearby metar is working now after the change.

I tough maybe you have know about the code of last rain on and the sunburn because your template and carterlake have different code for theses tags.

[WessexWeather]

Hello Wim,

Please would you look at my uvforecastv3.php script and tell me where I have gone wrong?

http://www.wessexweather.net/uvv3/uvforecastv3.php

Many thanks.

[wvdkuil]

Hello Wim,
Please would you look at my uvforecastv3.php script and tell me where I have gone wrong?
http://www.wessexweather.net/uvv3/uvforecastv3.php
Many thanks.

You use http://www.wessexweather.net/uvv3/uvforecastv3.php to start the script.
If you run the forecast with an url in the browser, the uvv3 folder is already in the URL so the setting at line 12 should be leaved as is in the download

Code: [Select]

$myfolder       = './';So change it back and the script should run fine.

If you include the script with php from inside another php script with

Code: [Select]

include './uvv3/uvforecastv3.php'the script needs to know where the images and cache-folder are located so the setting in line 13 should be used by removing the  comment mark # at the first position of line 13

Code: [Select]

$myfolder       = './uvv3/';
Wim

[Météo Québec]

Hi, can someone can explain or having an ideal why my
Weather data files    uploadWD/tagsWD.txt    NOT Current    0:05:43    0:05:30 10-02-2017 15:01
Weather data time    10-02-2017 15:00    NOT Current    0:06:45    0:05:30 10-02-2017 15:00


Thanks

[wvdkuil]

Hi, can someone can explain or having an ideal why my
Weather data files    uploadWD/tagsWD.txt    NOT Current    0:05:43    0:05:30 10-02-2017 15:01
Weather data time    10-02-2017 15:00    NOT Current    0:06:45    0:05:30 10-02-2017 15:00


Thanks

The statuspage is not for vistiors of your site, it is for you, the webmaster, to check the upload from Wd to your website.

I followed your status page for more then 15 minutes and there are only minor problems shown.

Normally the tags file is uploaded every 5 minutes. The status page checks the age and finds it is in your example 5 minutes 43 seconds old. It warns you that it is not within the set limit of 5 minutes.

The data inside the file has a time stamp also. The default settings allow the data to be 30 seconds older then the upload time. In your example the data in the file is 6 minutes 45 seconds old.

These values are very acceptable when they only happen a few times per day. Sometimes a server is very busy with other websites and the ftp processes have to wait.  All quite normal.

When there are multiple warnings every hour and especially when the age of the file and the the data is far more older , say 15 minutes, you should check the WD logfiles to see why the tags files are not uploaded frequently enough.

Wim

[Météo Québec]

Hi, can someone can explain or having an ideal why my
Weather data files    uploadWD/tagsWD.txt    NOT Current    0:05:43    0:05:30 10-02-2017 15:01
Weather data time    10-02-2017 15:00    NOT Current    0:06:45    0:05:30 10-02-2017 15:00


Thanks

The statuspage is not for vistiors of your site, it is for you, the webmaster, to check the upload from Wd to your website.

I followed your status page for more then 15 minutes and there are only minor problems shown.

Normally the tags file is uploaded every 5 minutes. The status page checks the age and finds it is in your example 5 minutes 43 seconds old. It warns you that it is not within the set limit of 5 minutes.

The data inside the file has a time stamp also. The default settings allow the data to be 30 seconds older then the upload time. In your example the data in the file is 6 minutes 45 seconds old.

These values are very acceptable when they only happen a few times per day. Sometimes a server is very busy with other websites and the ftp processes have to wait.  All quite normal.

When there are multiple warnings every hour and especially when the age of the file and the the data is far more older , say 15 minutes, you should check the WD logfiles to see why the tags files are not uploaded frequently enough.

Wim

Thanks Wim,

Can you tell me where exactly i can check this out?

Ex: FTP Log from WD?

[wvdkuil]

. . . shortened . . .

Thanks Wim,

Can you tell me where exactly i can check this out?

Ex: FTP Log from WD?

I am not running the 12 supported weather-programs anymore. But as far as I remember there was indeed an ftp log for the 5 minutes file.

The clientraw uploads use a separate ftp program with its own log, but there seems no problem with that.

Again 5 minutes upload always will have a few warnings every now and then, especially when it is busy on the server. On the computer-server your website is running, there are at least 10 other webservers for other domains asking for time also.  Especially FTP, having a lower priority and generating a lot of overhead, will have to wait when it is busy with normal HTTP website visitors.

You could set the upload in Wd to 4 minutes. that way there is an extra minute before you see a warning on the status page.

Wim