Topic: Importing 13 years / 1.26 million records into WL.com - feasible?

[Ray Proudfoot]

I’ve had a Davis VP2 ISS with separate anemometer transmitter for nearly 13 years. I run Weatherlink v5.9.3 on a Windows XP netbook. I cannot update to v6 because.NET 4.5.1 is not compatible with XP.

I’m interested in the WLL unit with a Pro subscription after learning it can import historical data.

But as I had an interval of 5 mins set (recently changed to 15) the export file from WeatherLink comprises 1.26 million rows.

From what I’ve learned the import into WL Pro requires a single import and as things stand no records can be edited once the data is imported.

Has anyone imported that much data and how well did it go? And how inconvenient is it being unable to edit ‘bad’ data?

I can’t understand why Davis didn’t permit multiple imports to lessen the dangers of a single huge import. Appreciate the thoughts of those who have done this.

[Randall Kayfes]

Greetings, I am in the exact same situation as you are Ray. To join for me would be 13 years of 5-minute intervals. I also would want to know the answer.

[WheatonRon]

Contact Davis support is what I would do.

[Ray Proudfoot]

Contact Davis support is what I would do.

I contacted Davis Support nearly a year ago over a different problem and never heard back from them.

The reason I’m asking here is because I would have thought some of you would have done this import and could report your experiences.

[Ray Proudfoot]

I decided to do what @WheatonRon suggested and write to Davis support. I received an email a few minutes later starting with this...

“Your request for technical support has been received and is being reviewed by our support staff.
During this holiday season we are receiving a high number of help requests...”

Seriously? What public holidays are there in the US on 1 February? If this is still the stock reply from Christmas/ New Year it’s extremely concerning. It suggests their support is still as bad as it was last year.

[DaleReid]

Ray:
Other than to say you did it, what value is there in having these data points in your current database?
Perhaps consider a year by year import and sealing each year at the end of the import, for historical purposes.

Your data are then available, can be manipulated by other types of more modern systems, even running on something like a Raspberry Pi 4 with a hefty SD card or a giant USB flash drive plugged in.  Custom reports and data pattern searches can be run without interfering with your live system.

Just because I have all my tax records since I started working doesn't mean I take them to the tax office every time I file?  I hate to sound skeptical, but I see a different need and solution for this historical data than having it readily accessible online.  Most people don't need to know, instantly, what the windspeed and direction was on 7 March 2013, for instance.

[DaleReid]

Ray, good point about the canned message you got from Davis on the higher number of requests for help.

I don't follow the history of manufacturers closely, but didn't Davis get bought up by some other firm?  I know Rainwise did and their support which had been remarkable years ago is pretty much non-existent.

The fact that they cannot update their web site or get a canned response changed with the passing of time isn't encouraging.

It's like getting the generic message that, even if you call some place in the night, you get a recording saying that due to higher volume of calls, it will be a bit before we get to you.  Always seems dismissive to me, and proper management at my company had daily review and updates on recordings to callers.  But that has also fallen by the side over the last two years so I guess it is a problem all over.

Dale

Oh, PS.  Recall that their vastly outdated display for the Vantage Pro and Pro 2 as far as across the room or other ease of visibility still hasn't been done in half a decade or more. 

[Ray Proudfoot]

Dale, if I didn’t import all my historical data it would be like starting afresh with a new station. Now I do have a website where the public can view current and historical weather and that will continue with Cumulus.

Your question about why would I need to know the weather on a certain day years ago well my website would reveal that but I would also like to see years of historical data in WL Pro.

This is perhaps why people have not imported historical data taken from a Davis PDF...

“A few steps are required to prepare the old data so that it is imported into WeatherLink Live correctly. Since this is a one-time effort, and failure to upload correctly is not easily undone, we urge you to use the PDF guide linked on the import page.”

So you can see it’s fraught with danger with little hope a second attempt is possible. You can read the whole document here. Note a one-time effort.

https://www.davisinstruments.com/blogs/news/import-your-historical-data-into-your-weatherlink-live

[DaleReid]

Your discussion is helpful.  Best of luck with your experiment.  I know that some of the big movers (NASA and all) have to be careful since they, too, only get one shot at it going right and hoping to not run into any unexpected things that would interfer with a successful import or command to 'fire retrorocket."

[Ray Proudfoot]

I’m beginning to understand what Jim Lovell felt like in Apollo 13. 

I’m holding back until I hear from Davis support or perhaps those here who have successfully imported and what precautions they took.

[Ray Proudfoot]

I have to say I’m astonished at the lack of response to this post. It looks to me as though the number of people who have bought the WLL haven’t imported any data. Maybe they just have the basic subscription which doesn’t allow for historical data.

Or maybe they’re biding their time because the import process looks fraught with danger especially when you consider it is recommended it’s done in a single pass that may involve hundreds of thousands of records or even more than a million like me.

Unless you have reasonable IT skills and can import into a spreadsheet or database how do you check for dirty data? Scrolling though a huge text file is impossible.

And still no response from Davis so even if I bought the WLL and paid for a Pro subscription I feel I would be left high and dry should I encounter a problem when importing the data. On that basis Davis is failing us big time. I’m holding off until such time as the import process is radically overhauled to make it far more user friendly and multiple passes are allowed.

[Ray Proudfoot]

Still no reply from Davis Support. The name seems like an oxymoron. Do they bother with customer enquiries from outside the North American continent?

[doubleohwhatever]

There has been a lot of consolidation in the consumer weather instruments sector over the past few years and Davis is included. They were bought by AEM.

Unfortunately, from what I've seen, a common outcome for most the these companies (Davis, Rainwise, Bloomsky, etc) after acquisition is a reduction in support staff. There also tends to be a "changing of the guard" in regards to company leadership and longtime employees that results in the remaining knowledgeable employees having a lot on their plates.

As for your particular problem, would it be possible to create a second weatherlink.com account to test the import and see what happens?

[Ray Proudfoot]

After much thought I have decided not to purchase the WLL or subscribe to the service. The lack of a response from Davis is only one small part of the reason.

My priority is my website which is available free to anyone in the world. It’s fed by data from Cumulus - a much loved application no longer supported but which remains stable if lacking sophistication for graphs etc.

I have added extra features to my website and I also download the dayfile.txt from Cumulus into an Access database for my own personal use. I can interrogate it in many different ways.

If I was using just the WeatherLink program designed around 30 years ago (and it shows) then WLL would be appealing.

But I feel I wouldn’t use its features enough to justify the cost and effort involved.

@doubleohwhatever, it’s sad what you report. Much the same as WUnderground which is a pale shadow of what it was 10 years ago.

[box]

My priority is my website which is available free to anyone in the world. It’s fed by data from Cumulus - a much loved application no longer supported but which remains stable if lacking sophistication for graphs etc.

I have added extra features to my website and I also download the dayfile.txt from Cumulus into an Access database for my own personal use. I can interrogate it in many different ways.

Perhaps you could at least upgrade Cumulus to Cumulus MX? With the webfiles etc you could get perhaps get your graphs?

[Ray Proudfoot]

Perhaps you could at least upgrade Cumulus to Cumulus MX? With the webfiles etc you could get perhaps get your graphs?

Not possible as my netbook is still running Windows XP. MX requires Windows 7 or higher. That would mean buying a second-hand laptop with an OS no longer supported. Too much effort involved and it would require a lot of work to adapt my website to MX.

The 13 year-old Samsung NC10 required a new fan two years ago and the HDD failed 10 months ago. I replaced it with an SSD meaning to two mechanical things that failed have been replaced. Should get a few more years out of it hopefully.

[box]

Perhaps you could at least upgrade Cumulus to Cumulus MX? With the webfiles etc you could get perhaps get your graphs?

Not possible as my netbook is still running Windows XP. MX requires Windows 7 or higher. That would mean buying a second-hand laptop with an OS no longer supported. Too much effort involved and it would require a lot of work to adapt my website to MX.

The 13 year-old Samsung NC10 required a new fan two years ago and the HDD failed 10 months ago. I replaced it with an SSD meaning to two mechanical things that failed have been replaced. Should get a few more years out of it hopefully.

That's a Shame. I wonder if you could go down the raspberry pi route?! They are pretty inexpensive and will run Mx on windows  or Linux. Maybe too much of a faff?

[Ray Proudfoot]

That's a Shame. I wonder if you could go down the raspberry pi route?! They are pretty inexpensive and will run Mx on windows  or Linux. Maybe too much of a faff?

That’s been suggested to me on the Cumulus forum but that’s an OS I’m unfamiliar with. Rather stick to Windows and Cumulus. It has no issues and with the lid down 99% of the time the power consumption is negligible. I have solar panels too. The basics are all displayed which is what me and my visitors are concerned with. My visitor map shows people all around the world have an interest. Ukraine and Laos being the more recent.

[Mapantz]

Ukraine and Laos being the more recent.

I guarantee that they are bots, Ray. 

[Ray Proudfoot]

I guarantee that they are bots, Ray. 

I link to my website on an international flight sim forum so it’s quite possible they’re genuine visitors.

Oh, and still no reply from Davis Support after a week. If you want the perfect example of an oxymoron that’s it! Useless. I won’t be bothering with their WLL.

[WheatonRon]

Today, I spent about 40 minutes on the phone with John in Davis support. My Weatherlink software, ver 6.0.5 went crazy and he carefully and methodically helped me fix the issue—in short, my data file got corrupted. My software was periodically sending CWOP, WU, MesoWest, etc., wind gusts of 327 mph so MesoWest appropriately called the reading “suspect”.  When I called Davis, a Thursday afternoon, I got right in—no hold time. I can’t be any more pleased. I should have asked John about email support but forgot. Ray, I know you live outside the U.S. so not sure how to help. But those of us who live here in the US—we are lucky when dealing with Davis support, at least on the phone.

Keep in mind, Davis was helping me with software that I paid probably $100 or so 15 years ago and such software is no longer being updated!

[Ray Proudfoot]

@WheatonRon, thanks for posting your experience with Davis Support. Phoning the US from England really isn’t an option. They appear to be treating customers who use email very differently to those who phone them.

Still no reply. I won’t be buying the WLL unit. Apart from the import process being so restrictive and no edit facility for any bad data the unit still appears to have issues.

[box]

Great avatar by the way!

[Bashy]

@WheatonRon, thanks for posting your experience with Davis Support. Phoning the US from England really isn’t an option. They appear to be treating customers who use email very differently to those who phone them.

Still no reply. I won’t be buying the WLL unit. Apart from the import process being so restrictive and no edit facility for any bad data the unit still appears to have issues.

Youre correct, emailing support is sporadic to say the least, can be days without a word.

Dont let that put you off WLL though, its not a bad unit, I'm not impressed with their Mobiize using a web browser though, the rain amount is still showing as inches when it should be in mm, they've known this since October.....

[Ray Proudfoot]

@Bashy, having given it considerable thought even if I managed to import 13 years of data into my account what the software offers me over what I have with Cumulus is simply not worth the money or effort.

My email to support last April on another topic was never answered so this isn’t a one-off. It appears to be a policy for overseas customers. Their kit is great. Their support is non existent as far as I’m concerned.