Auto Scrape data from WL ip screen? RESOLVED ish

[johnd]

OK, so that we've got some frame of reference:

0. (This should have been Q1 but I forgot it!) Have you done a download of the latest archive file from weatherlink.com? If you look in Browse mode within the Weatherlink program then can you see all of the latest records?

1. What's your skeleton.htx file actually called?

2. Are you using Profile 1 (or if not then which profile)?

3. What do you have the 'Upload every' set to in the profile?

4. Do you have 'Local Transfer Enabled' set to ON? If yes then can you see skeleton.htm created in wherever you have the local path pointing to?

[m77]

0. (This should have been Q1 but I forgot it!) Have you done a download of the latest archive file from weatherlink.com? If you look in Browse mode within the Weatherlink program then can you see all of the latest records? CHECK

1. What's your skeleton.htx file actually called? records.htx

2. Are you using Profile 1 (or if not then which profile)? yes - profile 1

3. What do you have the 'Upload every' set to in the profile? 5 mins

4. Do you have 'Local Transfer Enabled' set to ON? If yes then can you see skeleton.htm created in wherever you have the local path pointing to? yes, it uploads to a folder on my PC and looks the same as the link above (no data)

[m77]

Solved - i was basically editing the html file. i then discovered what the proper .htx file looked like and i edited the template one to suit my needs.

thanks once again guys.

[m77]

Is there a tag that does highs and lows of all time, rather than just yearly?

(by the way, my page that is being tweaked is http://users6.jabry.com/maidenerleghweather/final.htm )

[johnd]

While you've got the WL program running, don't forget to look at the NOAA reports for the generation of day-by-day reports in the current month.

Just to reiterate: If you choose Reports | Summarise Month... you'll find that you will be able to save the day-by-day report to a text file (called noaamo.txt by default), which you can then upload to your website. (Most browsers will display a .txt file just as they will a .htm file, although the .txt file will obviously look rather plain. So all you need to do is do add a link from your school's weather home page to noaamo.txt (which would really be better renamed as eg jan2010.txt or some such).

[m77]

Brill. I am looking at that now.

Is it possible to display all time records rather than just yearly?

Or perhaps display the yearly figure if it is larger than a value I enter?

[dalecoy]

Is there a tag that does highs and lows of all time, rather than just yearly?

According to Davis, the tags listed in the document that is installed with WeatherLink, are the only ones that exist.  And I know of none that have been discovered that are not in that document.

So, you'll have to write a program that captures the file late on December 31st, and ......

Of course, you could always lobby Davis to add more tags.  For instance, I've been asking for the past 3 years for "current THSW".  You can see how far that has gotten.

[dalecoy]

(by the way, my page that is being tweaked is http://users6.jabry.com/maidenerleghweather/final.htm )

I find it interesting to see the maximum rainfall intensity in mm/hr (metric),

but the strongest wind gust in mph.

[m77]

yeah, in the UK (i don't know where you are) we measure in metric and teach the kids in metric...... but drive in miles, so most think about mph rather than kmh. Mixed systems i know.

We actually teach the kids to use kmh for mapwork yet the road signs are all miles.

[m77]

So, you'll have to write a program that captures the file late on December 31st, and ......

can you elaborate on this please dale?

sounds good.

[dalecoy]

yeah, in the UK (i don't know where you are) we measure in metric and teach the kids in metric...... but drive in miles, so most think about mph rather than kmh. Mixed systems i know.

We actually teach the kids to use kmh for mapwork yet the road signs are all miles.

I'm in the USA (state of Missouri, actually).  We get occasional criticism for not exclusively using metric - but I think using a mixed system would be even more confusing.  

In general, "scientific" work in the USA is done in metric, and would express speeds in metric units.  However, "popular" work, like weather reports, is always done in english units.  So, rainfall in inches, wind in mph, etc.  [But of course temperatures in Fahrenheit].

But I quite agree that the presentation should be in the units most conventional for the audience.

[dalecoy]

So, you'll have to write a program that captures the file late on December 31st, and ......

can you elaborate on this please dale?

sounds good.

OK, I don't know how much (if any) computer programming or scripting you know, but let me outline the general principle.  Again, I suggest that one of your talented kids could put this together.

1.  The .html file that's uploaded every few minutes to your web site, is really just a text file with mostly-consistent strange things for formatting and titles, interspersed with a few characters of data.  [As a convenience, a copy of that file is usually present on your computer, as you have noted]

2.  It's easy enough to schedule a computer program to run at 23:59:00 on December 31.  The program could do something as simple as making a copy of the file to somewhere else on your computer - perhaps with a slightly different name (whatever-2010.html) so that it won't be destroyed by overwriting later.  And/or print it. 

3.  Now, of course, you could just read that file sometime later (New Years day or...) - or display it using a browser.  And manually compare with your web page that displays the "forever extremes", and change those that were higher/lower last year. 

You could do all of that yourself (perhaps with a tiny bit of assistance).

4.  But it's a rather simple programming task (if you already know some programming) to do Step #3 using computer code.

Footnote: that's essentially what screen scraping does.

Footnote 2: of course, that just does "forever extremes for past years".  You would really want to adjust the extremes when this year's extremes were higher/lower.  In the "extreme", that would require looking at every data file and usually not changing any extreme. 

[Bushman]

I have some simple Autohotkey pgms. that can do some (most?) of what you want.  Dale is right - you need to start by scheduling  a yearly DL.

[m77]

I am pretty pleased with this Excel sheet that i made - so the pupils can view data realtime, WITHOUT ME having a computer running 24.7.365

check it out!

http://www.school-portal.co.uk/GroupDownloadFile.asp?GroupID=910590&ResourceId=2781157
i am particularly chuffed with the wind rose. the screen reminds me a little of the Weatherlink bulletin screen!

[dalecoy]

Very nice.

I assumed that the link was "live", not static.

I clicked that link at 4:46 PM (local in Missouri, USA) - but it says "Data correct as of 9:55 Friday, January 15, 2010". - I can't figure the discrepancy between :46 and :55 - and  9:55 is not identified as to AM/PM. 

Also, perhaps you should clarify "This page refreshes every minute" - because I can't see it refreshing (using Excel, at least).

On the other hand, if it's a static page, please ignore the comments above.

[m77]

it is meant to be live!

did you 'enable macros' when offered the chance on opening?

it updates every 1 minute for me.

can anyone else get it to work and auto refresh or not refresh? please post below.

I have also changed it so it shows either AM or PM GMT. make sure you click on the new link above.

[dalecoy]

Thanks for adding "AM or PM GMT"- but something is still wrong there.  I just opened your new link, and it shows data correct as of

7:36 Saturday, January 16, 2010
am GMT

But I'm in the Central Time Zone of the US, which is UTC - 06:00, and it was (when I opened it) just 2 hours time difference (i.e., about 9:36 AM).

===========

Re: the updates - I'm using Excel 2007 on a Windows 7 system, and it appears that some sort of security settings are completely prohibiting me from enabling data updates (unless I unconditionally enable them for everything from anybody, which I'm reluctant to do).  So, that's apparently just a problem on my end of the link.

[m77]

Don't know mate. It should refresh on opening and also every minute.

it works perfectly for me at home and at work.

Do you get an option about macros when you open it?

You are using excel to open it?

Can someone else try please?

[Bushman]

Tried on Excel 2003 - macros enabled. Works fine.  FWIW, the graphs overlap a bit.  You might want to DL the data to a separate workbook while you are at it.  Did you use VB?

[m77]

No I used formulae.

Why dl to another book?

[m77]

No I used formulae.

Why dl to another book?

[Bushman]

Well, the raw data can be easily transformed/post processed unlike the graphs.  Would make a good exercise for students.  And because there are better graphs to represent your data than the ones you have used IMO.

[m77]

The data is on another sheet. Why does being in another book make a difference?

And which graphs in excel do you mean?

Thanks.

[Bushman]

My pref is to have the data on one sheet, the graphs on another in the same workbook.  YMMV    Things like the last 7 days of temp would be best shown via line graphs  or if you are looking for discrete days, usse a candlestick chart.

[m77]

i agree Bush with your choice of graphs, but i think you have misunderstood the book.

the workbook exports the WL.com page once you open it - it does store or scrape or log the information at all - it is merely a graphical representation of current conditions. the Wunderground graphs on my site do the things that you suggest.

I also agree that line graphs are best for continuous data, but that is not what is shown here.

Thanks again bud.