Topic: VWS and METAR report

[ohlone]

HI

OK so I created the perl script to pull the data,  I tried the Https location,  that seems to not generate a useable file format for VWS. 

However,  I switched back to the ftp site and its working properly.

Here is the code for the perl script I used,  Its not optimized in any way.

 [ You are not allowed to view attachments ]

[Blizzard]

Ok! Thanks for sharing. And where do you put that script into action? How does it know how to get the right sky (icon) condition? Thanks again! I probably don't have the same level of knowledge.

[ohlone]

Use the task scheduler in windows to run it every 15 mins.

In vws you add the metar icons to the desktop and tell them to generate jpeg file

I forget the exact commands but the manual covers that

[ohlone]

You will need to install perl64 if you don’t already have it

[Blizzard]

I don't have it. I will get it. My issue or question is more as to kmow how or why the .TXT file becomes available by VWS. When I copy and paste the file from NOAA folder, it doesn't work. So, I wonder why and how the glitch you propose can work. I will try it later as I am done for today trying to fix that issue. And is the Icon generator for your own staion has access to the different Metar stations in wvs/metar if you lock it it on one specific station?

One thing I have noticed this afternoon beweent the ftp and http site is that the time on the creation of the files fit, but the content in it evolves on the http site and not on the ftp site if I copy and paste de two sation.TXT files. They look the same in form, but not in content. Isn't weird? It is as if ftp site uploads the information of the file itself without the content or really udates one or twice a day as I've noticed so far.

[Blizzard]

Here are the 2 files. Open them.

[ohlone]

I think windows ftp is caching the files.  I’m going to try to delete cached .txt file before loading to see if it helps.

[Blizzard]

So? And what do I do after downloadin perl64?

[ohlone]

Create a task in windows task scheduler to run the Perl script every 15 mins.  So far I’m still fighting the windows caching - I still havent solved it.

[Blizzard]

So, it is not working yet? On my side, I think I Will use the AI to generate a code that displays my set of icons on my website according to the conditions of the station I want that is frequently updated on the NOAA https site. I guess that is possible. It would finally be a a first and good use of the AI lol

[Blizzard]

Well... nothing really works on my side. I wrote to NOAA asking why the file I am looking for is updated once a day on their ftp site as opposed to every hour on their https site.

ftp://tgftp.nws.noaa.gov/data/observations/metar/stations/

https://tgftp.nws.noaa.gov/data/observations/metar/stations/ 

I am hoping for an answer and that they will be able to make sure the two files are available at the same time.

I tried some Python scripts, but generating the right icon according to the metar obs and looking in the right folder (day/night) is not easy. For now, I retrive the .txt file on their https site and put it in the vws/metar, but VWS won't read it unless I shut and restart it. Do you have an idea on how to force VWS to read the file without reguraly shuting it down throut a Windows commande?

[ohlone]

Are you sure FTP is not updating?  I was wondering if it seems that way due to windows caching fTP files.   If it’s really the case FTP only updates daily that’s a deal breaker.  Try rebooting your pc (flushes the cache) before loading ftp I’ll bet you see updates.  I’m away for a couple days I will try winSCP next instead of Perl. 
BTW VWS has a “FTP download “ section but I get error retrieving file trying to point it at the noaa type site.   It works on the http site… go figure.

[Blizzard]

Actually, I rebooted my computer for other purpuses. For a while also, https was note working as if they were performing stuff. I was getting an error message. Then, everything seems to be back to normal. I have noticed two updates in a row of the same file which I hadn't for the last couple ou days. So, As for now, VWS seems to be working juste fine with NOAA (FTP). Is it the cache or the email I wrote to them, I don't know. And when you are talking about the FTP cache, whre and how do we empty it? Thanks a lot for your help.

[Blizzard]

UPDATE ON MY STITUATION

I see that the file on NOAA FTP updates every hour, but when I copy and paste the file, I get the old data. As if I was downloading an empty egg. The shell of the file looks fine and new, but I just get old stuff. When I reboot my computer, then I get a perfect file but again, the other ones look the same. It downloads, but no new data. So, you are 100% right concerning the FTP cache. So,

1) We figure out how to empty the cache or bypass that process.
2) We reboot computer every hour. It doesn't make sens but it would work.
3) We retrieve the .txt file from the NOAA https every hour around the update time and restart VWS with a off and on Windows task in order VWS reads the NOAA.TXT file.

[ohlone]

Download winSCP it is open source, you can create a command line batch file to download the files , then use task scheduler to run the process regularly.  I’m hoping winSCP is able to get around the stale cache problem in windows. 
Rebooting or restarting hourly is not ideal. But I think wvs has a setting to reboot itself automatically if you want to try it.

[Blizzard]

Well, the command to download it isn't a problem in itself. VWS has that FTP retrieve fonction that works if I download it from https. Downloading it from ftp with a third part software will work too. The problem in that case that remains is having VWS create the csv file and read it. The only way to do so has been getting the file from FTP site with VWS or put the file in the vws/metar folder and restart VWS. You are fight, VWS as that reboot computer function, but it would have been better to be able to select computer or software with that thing. You are right. Rebooting every hour... no way.

[Blizzard]

Still figuring a way out. If I come back to your perl script, it can read the sky and weather condtions of the metar and create the image?

[ohlone]

I found that over time VWS stops creating the CSV files and gives errors .

The script seems dependent on windows ftp so cache is a problem
It would need to switch to http.  Now a good question is if we could create .CSV files
Basically replacing the whole metar download from VWS.  Then hopefully the JPEG file creation is independent and can use those csv files. 
If it’s a simple thing to add all the commas between the entries we might try that route.
Creating jpegs from the script itself is probably too complicated.

[Blizzard]

The cache is a problem, but also the date of the file. On the https site, it is in zulu while it is in eastern time on the ftp. Also, it doesn't seem that VWS gets anything from the NOAA site. It doesn't seem to be looking in the right folder. The only moment it seems to do so is when we put the ftp file in the folder through Windows ftp dowload. After, it only generates that same file over and over. I tried getting the ftp file from another computer after cleanning the disc and emptying the folder cache of my main computer running VWS, the software won't read that file at all when updating METAR. The only way I've found so far is shutting down VWS and starting it ofter putting a new .txt file in the folder either from ftp site ou https meaning VWS can read both files, but cannot download ftp on a regular basis. Event trying to retreive the file from VWS trought ftp retreive instead of the NOAA function will not work. In other words, it's like we need to open the DOOR through WINDOWS (Ah! Ah!) to connect to NOAA FTP, but then, the file gets stuck in the cache. It seems to me that after 24h, it succeeds at downloading a new file as if the date add something to do tu bypass the cache.

[ohlone]

Made good progress today. Will post update later tonight. 
Basically I created a batch file that uses WINSCP to grab the files and put them in the metar folder.
Then it kills VWS.exe waits 20 sec then restarts it

I used task scheduler to run this every hour

I’ll post the script later. 

I turned off metar updates to not waste time trying to access the wrong site .

[ohlone]

Here is the script.  I start it hourly at XX:58 so there is time to complete the download , reboot, and update the jpegs on the hour. you just need to run the program "c:\vws\WINSCPScript.bat"  for the task.  Modify it to choose the METARS you want to get from nooa.

It appears VWS will update the csv files from boot up, without trying to access ftp.  It must be part of the initialization code.  The csv updates get skipped if there is error downloading the files, which I could not get to be consistent. So I concluded reboot at the right time is needed.  I researched how to do this in batch file and you can see the sequence. I made the reboot to happen after the Metar files are loaded.

The WINSCPScript.bat is run close to the hour to pull the .txt files right after the METAR updates. And I have no cache issues with winscp. IN fact it preserves the timestamps on the file when it copies to your local directory.  It has a logging function so you can see what it does. Ill probably add a line to to delete the Log as part of the script.

I got the script parameters using the WINSCP GUI and then cut and paste my commands.  So no Perl script is needed.  just runs as .bat file.

You can see the icons updating hourly at about 10 mins after the hour at:
http://ohlonetrail.net/wx.htm

Of course you need to install WINSCP.exe which seems pretty much free to use.


BTW I checked Aerisweather  (which took over hamweather) .  Lowest cost is $300 per month.  SO you can forget using that option!.

[Blizzard]

I will check that this weekend. I was going that way too. Do we need to download anything with your script or it works with Windows? Also, I run virtual VP to split the data with another software and I have noticed VWS will communicate better with VVP when the latest has just openned. So, can I modify the script to shut down and restart both softwares?

[ohlone]

Works with Windows and yes you can modify as you need.
I ran in to the problem that "Taskkill"  leaves the VWS Tray icon and keeps adding more after each restart,  until you mouse over the area.   I found a fix for that which updates registry settings and reboots explorer.  This "cleans" the dead icons in the tray.  I've added the steps to the script.  Updated and attached.

Another item - I allow VWS to reboot the computer once daily at 2:55am.  I modified task scheduler to run the script with a delay of 3 minutes, so its pulling the Metars at about 2:58am and every hour after.
This gives VWS enough time to get ready and update the icons on the hour.

The daily reboot keeps the timing of the script in perfect sync.

[ohlone]

Man all the "hacks"  needed to work around the fact that there is no way to change the URL for NOAA FTP site   

Programmers please don't hard code URLs....

[saratogaWX]

Well, VWS was last updated in 2015 and the bulk of the support DLLs were from 2009.  It's not really a surprise that 9 years later, something would break in an un-maintained program.

I strongly suggest you both consider switching to supported weather station software (like CumulusMX) -- my own VWS (test for the templates) frequently 'goes to sleep' so I can't depend on it to keep a website current.  Just my thoughts...