I want to go to the next step. WL.com ip to WD live. RESOLVED.

[m77]

I have weatherlink ip that runs without the WL software running 24.7
I have managed to scrape basic data from my WL.com page to produce a sticker. click on my banner to see what I have done.

I have worked out a way to use the scraped info from the WL.com summary screen to create a clientraw.txt file if required. obviously not the full version but using any statistic from the WL.com screen.

my question is where can I go now?

I am not able to run software 24.7 so that is a limiting factor that cannot be compromised.

any suggestions on what I could do now? web templates? php? any ideas welcome.

[timmcmanus]

What do you want to do?

[m77]

well I would like to improve my web template.

I have looked at the saratoga script which requires clientraw.

I have created a php script that scrapes my data and creates a fake client raw that contains basic data.

as some data is not present eg historical stats, what can people recommend that I use to improve my web interface with the public.

I manually run wxsim once an
 evening.

[johnd]

I'm afraid that I don't think that there's any simple short-term solution here. That's not to say that it's impossible to offer various ideas, but they're all potentially in the future or need some development work. For instance:

1. I'll be very surprised if Davis don't have some enhancements soon for the weatherlink.com website. I've not heard of anything specific about to happen but a lot of wl.com users must be clamouring for more features so it would be an obvious extension of the appeal of wl.com if Davis could offer more functionality. Historical graphs would be one pretty obvious new feature - they've obviously got all the necessary data just sitting there in each user's archive data.

2. And then building on the idea of the archive data being online at wl.com, it's again an obvious extension for users to download this data automatically to their own PCs to maintain a complete historical database there. For example if you were to run a a live copy of WL at home and set up a web station with hourly automatic downloads of archive data you can have a plot graph open which updates automatically every hour. This works fine but there are currently two gotcha's:

The first is that the Internet Settings options are greyed out for web stations so you can't use this approach to maintain a website presentation - you can only see the display on your local monitor.

Second the archive data is only uploaded to wl.com once an hour and so your Plot graphs will typically be at least an hour out of date.

The only other way that I know of for getting hold of the archive data is that the WL DLL downloadable from Davis does now have functionality for accessing web station data. So in principle you can write your own Windows software that includes the DLL and then  could plot the downloaded archive data to your graphic requirements and upload it further to a web server as required.

3.  There are of course several miniature PCs like MeteoPlug, Bifferboard etc that are so small and inconspicuous as to be indistinguishable from a small plug-top power supply or similar. These don't look anything like a PC. You might be able to get away with running one of these 24/7 at the school - if you can run the WLIP logger itself then why not a Bifferboard for instance - there's really no difference in principle in what they're doing. But AFAIK they might be better suited to a serial or USB logger than a WLIP one (or at least a WLIP input might require further development).

4. Write some software that stores the data scraped from the current conditions screens in a database and then manipulates and graphs it according to your own specification. There actually isn't anything too difficult in doing this for anyone with some basic programming skills, although it obviously wouldn't be a 10-minute project. But if you've never done any real programming then it's probably not an option.

[d_l]

Get a SheevaPlug with Meteohub hub on it.  The Meteohub firmware can make graphs and collect the WeatherDisplay data, e.g. the various clientraw files so you could use WeatherDisplay Live on your web site.  If you are using a Meteohub, the WeatherLink software is not as easy to use with the WLIP, but can be used if you are willing to accept occasional download failures.

[m77]

thanks guys.

john's option 4 interests me.

i currently upload the daily data every evening to my host.

i have basic php scripting skills and have written a code to calculate various stats. obvious it only updates when i upload the daily data and is not live.

with your idea john - i fancy a crack.

would it not need a script running 24.7 to scrape the data to send to a database?

any tips on a start?

[wuhu_software]

"The only other way that I know of for getting hold of the archive data is that the WL DLL downloadable from Davis does now have functionality for accessing web station data."

John,

Are you reffering to the following function to communicate with the WL IP?

short int OpenTCPIPPort_V (const char *TCPPort, const char *IPAddr)

There is not a method for communicating with wlink.com to retrieve data is there?

Thanks.

[johnd]

Are you reffering to the following function to communicate with the WL IP?

short int OpenTCPIPPort_V (const char *TCPPort, const char *IPAddr)

There is not a method for communicating with wlink.com to retrieve data is there?

No, I was thinking of:

int DownloadWebData_V(DateTimeStamp dateTimeStamp, char *userName, char *password)

Must admit that I've done nothing more than make a mental note that this new command has been introduced in the DLL v2.42 (though the DLL versioning/documentation doesn't seem fully consistent - I'm not sure that this isn't actually v2.43). I keep meaning to find time to have a play but that opportunity has eluded me so far this year. So I can't speak from personal experience of having used this command.

[johnd]

Get a SheevaPlug with Meteohub hub on it.

MH looks like a good system but I'm not sure in detail how well it plays with uploads to wl.com when the data source is the WLIP. Has anyone here tried it? Is it clever enough to time the loop requests so that the uploads to wl.com still continue successfully (both current conditions and archive uploads)?

[m77]

is there a way in php or other to send scraped data to a file WITHOUT any script being run?

so the hosted file knows somehow to collect data every 15 mins for example.

[wuhu_software]

Is there a way to have Weatherlink upload to the Weatherlink.com site or is this strictly for WeatherlinkIP?

It seems easy enough to retrieve the data but I do not own a WeatherlinkIP.

Thanks.

[d_l]

MH looks like a good system but I'm not sure in detail how well it plays with uploads to wl.com when the data source is the WLIP. Has anyone here tried it? Is it clever enough to time the loop requests so that the uploads to wl.com still continue successfully (both current conditions and archive uploads)?

John, I wouldn't be recommending it if I wasn't running one.  In fact, my WeatherPlug (SheevaPlug Meteohub from RainmanWeather) is concurrently pulling data from two WLIPs connected to two Envoys gathering data from two different VP2s (I wouldn't recommend trying this load on a NSLU2-version Meteohub though with it limited processor power).  You can see the results on my web site.   The second VP2 is a neighbor's station that is mounted higher than mine and so has different wind readings. On occasion, the WeatherLink.com current condition uploads have a few minute delays between updates because there is a lot of access activity on the WLIPs.

Boris has steadily improved the firmware so that it handles the WLIP current and archive uploads much more smoothly.  Earlier versions worked pretty well, but the latest version also seems to have improved WeatherLink software archive downloads from the logger.  These downloads are still somewhat hit-or-miss as to conflicts with the WLIP uploads and the Meteohub LOOP activity, but the conflicts have been reduced.

A month or so ago there was a problem where the Meteohub would "sort of lock up" waiting for the WLIP uploads to finish after connecting, but as best as I could determine that was caused by the Davis site not allowing the uploads to complete (this was shortly after the Davis site's server problems).  The Davis site is probably the weakest link in this arrangement.

[m77]

It seems easy enough to retrieve the data but I do not own a WeatherlinkIP.

Thanks.

do you mean retrieve data from WL.com?

if so how would you do that without the software running, or is it not possible?

[johnd]

John, I wouldn't be recommending it if I wasn't running one...

Sorry - didn't mean to imply otherwise - just wasn't 100% clear from what you posted. And although WLIP is mentioned on the MH wiki/documentation, I couldn't quickly spot where the details of what's possible were spelled out.

So MH sounds like quite a good solution, at leasting in meeting some sets of requirements.

[m77]

if i can get a php script to get the data i need from WL.com summary pages, would a CRONJOB work - in that it would trigger the script to run, collect the data and add it to a database?

i.e. can i set up a CRONJOB and then switch off my pc and forget about it?




EDIT : just found out that cron is something that is on and run by my host server.

so i could set up a cron command to run a php script that scrapes the data I want and adds it to a database file on the server. (perhaps in the form of a clientraw file.)

if this does work, then wouldn't templates that need clientraw, also work?

if not, surely it opens the door a little more for folks who want more from WL.com without 24.7 PC operation, by allowing an accessible database to be built up ?

[johnd]

m77, I'm not a user of PHP so let me leave that answer to someone else. But the key point is that some computer somewhere obviously needs to be running 24/7 to service that data acquisition from wl.com and to prepare new web-page graphs etc all the time.

But any suitable computer will do. It could be a tiny SheevaPlug or BifferBoard. It could be your own PC. Or it could be a server somewhere/anywhere or just one of these rather nebulous cloud computing services, if one were set up. It just needs to be a computing platform/service somewhere with Internet access and which supports programs or scripts written in a language that you can author. (Or am I just stating the obvious here - apologies if I am!)

[m77]

thanks john.

you say that a computer is needed to generate the graphs etc, but AFAIK, if my cron idea will work the only pc is the one used by the end point viewer, that is once the php script, csv data file and cron command are set up.

surely its a winner if I am correct about cron?

[m77]

..... as the pc you speak of is my host server.

[wuhu_software]

It seems easy enough to retrieve the data but I do not own a WeatherlinkIP.

Thanks.

do you mean retrieve data from WL.com?

if so how would you do that without the software running, or is it not possible?

Yes, you would need some type of PC to retrieve the data. This would be easy to do with a low-power consumption Windows machine. Achieving this on a different platform like an embedded Shevaplug would require more work unless Davis supplied a DLL or library for Linux. Short of supplying a library, they would need to release their protocol information required to talk to their server. I would not hold my breath on that because incorrectly written code could interfere with the operation of their servers.

[m77]

thanks mate.

can any one comment on my cron job idea?

[johnd]

...This would be easy to do with a low-power consumption Windows machine. Achieving this on a different platform like an embedded Shevaplug would require more work unless Davis supplied a DLL or library for Linux....

But doesn't MeteoHub, at least in a sense, do this already? d_l will probably be able to provide more detail, but AIUI the configuration would be that the WLIP plugs into the LAN, which also has Internet access for routing the wl.com uploads. Then the MH device is also plugged into that same LAN and talks direct to the WLIP to acquire its live LOOP data. The MH device then processes your data according to how you have it configured, eg clientraw.txt generation for uploads to a WDL server somewhere. Apparently this all works successfully.

[wuhu_software]

I believe, and I may be wrong, Davis does not allow any device other than WeatherlinkIP to upload to their site, not even the WeatherLink software running on a PC. Perhaps that is incorrect, but that is my impression.

If Davis allowed anyone to upload from whatever device they wanted, this could create a support nightmare for them with no profit to be had.

[johnd]

I think we're at cross purposes here. I agree with what you're saying - only the WLIP logger can (for now at least) upload to wl.com.

But the configuration I was talking about was where the WLIP logger itself continues to upload its data to wl.com while another IP device (a MH device in this instance) requests a LOOP data stream from the WLIP logger and generates additional text or graphic files. To repeat, AIUI this configuration seems to be abloe to work perfectly well.

You could - if you so chose - embed presentations from both sources on the same web page, eg an iframe looking through to one of the wl.com presentations plus eg another iframe/whatever hosting a Weather Display Live presentation, driven by a clientraw file generated by the MH device.

[wuhu_software]

Meteohub could add the support you are looking for since they already do support the WeatherlinkIP.

Whether not they already support the file type you are looking for, that is another question.

[m77]

and cron?

or noone sure and leaving the proof of the pudding with me?

will experiment on Sunday.