Author Topic: Importing 13 years / 1.26 million records into WL.com - feasible?  (Read 1661 times)

0 Members and 1 Guest are viewing this topic.

Offline Ray Proudfoot

  • Forecaster
  • *****
  • Posts: 446
    • Cheadle Hulme Weather
Iíve had a Davis VP2 ISS with separate anemometer transmitter for nearly 13 years. I run Weatherlink v5.9.3 on a Windows XP netbook. I cannot update to v6 because.NET 4.5.1 is not compatible with XP.

Iím interested in the WLL unit with a Pro subscription after learning it can import historical data.

But as I had an interval of 5 mins set (recently changed to 15) the export file from WeatherLink comprises 1.26 million rows.

From what Iíve learned the import into WL Pro requires a single import and as things stand no records can be edited once the data is imported.

Has anyone imported that much data and how well did it go? And how inconvenient is it being unable to edit Ďbadí data?

I canít understand why Davis didnít permit multiple imports to lessen the dangers of a single huge import. Appreciate the thoughts of those who have done this.

Offline Randall Kayfes

  • Weather - Photography - Astronomy - Computer Admin
  • Forecaster
  • *****
  • Posts: 1944
    • Arizona Kaymann
Re: Importing 13 years / 1.26 million records into WL.com - feasible?
« Reply #1 on: January 31, 2022, 07:16:31 PM »
Greetings, I am in the exact same situation as you are Ray. To join for me would be 13 years of 5-minute intervals. I also would want to know the answer.



Offline WheatonRon

  • Forecaster
  • *****
  • Posts: 1187
    • WUnderground
Re: Importing 13 years / 1.26 million records into WL.com - feasible?
« Reply #2 on: January 31, 2022, 09:47:03 PM »
Contact Davis support is what I would do.
Davis VP2 with SHT31 (3 complete VP2 systemsó2 with a daytime fan and 1 that has a 24 hour fan); CWOP--CW5020, FW3075 and FW4350; WU--KILWHEAT17, KILWHEAT36 and KILWHEAT39; WeatherCloud.net; CoCoRaHS--IL-DP-132; and Weatherlink 2.0

Offline Ray Proudfoot

  • Forecaster
  • *****
  • Posts: 446
    • Cheadle Hulme Weather
Re: Importing 13 years / 1.26 million records into WL.com - feasible?
« Reply #3 on: February 01, 2022, 03:24:33 AM »
Contact Davis support is what I would do.

I contacted Davis Support nearly a year ago over a different problem and never heard back from them.

The reason Iím asking here is because I would have thought some of you would have done this import and could report your experiences.

Offline Ray Proudfoot

  • Forecaster
  • *****
  • Posts: 446
    • Cheadle Hulme Weather
Re: Importing 13 years / 1.26 million records into WL.com - feasible?
« Reply #4 on: February 01, 2022, 12:09:25 PM »
I decided to do what @WheatonRon suggested and write to Davis support. I received an email a few minutes later starting with this...

ďYour request for technical support has been received and is being reviewed by our support staff.
During this holiday season we are receiving a high number of help requests...Ē

Seriously? What public holidays are there in the US on 1 February? If this is still the stock reply from Christmas/ New Year itís extremely concerning. It suggests their support is still as bad as it was last year.

Offline DaleReid

  • Forecaster
  • *****
  • Posts: 1943
    • Weather at Eau Claire, WI
Re: Importing 13 years / 1.26 million records into WL.com - feasible?
« Reply #5 on: February 01, 2022, 12:59:22 PM »
Ray:
Other than to say you did it, what value is there in having these data points in your current database?
Perhaps consider a year by year import and sealing each year at the end of the import, for historical purposes.

Your data are then available, can be manipulated by other types of more modern systems, even running on something like a Raspberry Pi 4 with a hefty SD card or a giant USB flash drive plugged in.  Custom reports and data pattern searches can be run without interfering with your live system.

Just because I have all my tax records since I started working doesn't mean I take them to the tax office every time I file?  I hate to sound skeptical, but I see a different need and solution for this historical data than having it readily accessible online.  Most people don't need to know, instantly, what the windspeed and direction was on 7 March 2013, for instance.
ECWx.info
&
ECWx.info/t/index.php

Offline DaleReid

  • Forecaster
  • *****
  • Posts: 1943
    • Weather at Eau Claire, WI
Re: Importing 13 years / 1.26 million records into WL.com - feasible?
« Reply #6 on: February 01, 2022, 01:04:16 PM »
Ray, good point about the canned message you got from Davis on the higher number of requests for help.

I don't follow the history of manufacturers closely, but didn't Davis get bought up by some other firm?  I know Rainwise did and their support which had been remarkable years ago is pretty much non-existent.

The fact that they cannot update their web site or get a canned response changed with the passing of time isn't encouraging.

It's like getting the generic message that, even if you call some place in the night, you get a recording saying that due to higher volume of calls, it will be a bit before we get to you.  Always seems dismissive to me, and proper management at my company had daily review and updates on recordings to callers.  But that has also fallen by the side over the last two years so I guess it is a problem all over.

Dale

Oh, PS.  Recall that their vastly outdated display for the Vantage Pro and Pro 2 as far as across the room or other ease of visibility still hasn't been done in half a decade or more. 
ECWx.info
&
ECWx.info/t/index.php

Offline Ray Proudfoot

  • Forecaster
  • *****
  • Posts: 446
    • Cheadle Hulme Weather
Re: Importing 13 years / 1.26 million records into WL.com - feasible?
« Reply #7 on: February 01, 2022, 01:35:44 PM »
Dale, if I didnít import all my historical data it would be like starting afresh with a new station. Now I do have a website where the public can view current and historical weather and that will continue with Cumulus.

Your question about why would I need to know the weather on a certain day years ago well my website would reveal that but I would also like to see years of historical data in WL Pro.

This is perhaps why people have not imported historical data taken from a Davis PDF...

ďA few steps are required to prepare the old data so that it is imported into WeatherLink Live correctly. Since this is a one-time effort, and failure to upload correctly is not easily undone, we urge you to use the PDF guide linked on the import page.Ē

So you can see itís fraught with danger with little hope a second attempt is possible. You can read the whole document here. Note a one-time effort.

https://www.davisinstruments.com/blogs/news/import-your-historical-data-into-your-weatherlink-live

Offline DaleReid

  • Forecaster
  • *****
  • Posts: 1943
    • Weather at Eau Claire, WI
Re: Importing 13 years / 1.26 million records into WL.com - feasible?
« Reply #8 on: February 01, 2022, 02:00:49 PM »
Your discussion is helpful.  Best of luck with your experiment.  I know that some of the big movers (NASA and all) have to be careful since they, too, only get one shot at it going right and hoping to not run into any unexpected things that would interfer with a successful import or command to 'fire retrorocket."
ECWx.info
&
ECWx.info/t/index.php

Offline Ray Proudfoot

  • Forecaster
  • *****
  • Posts: 446
    • Cheadle Hulme Weather
Re: Importing 13 years / 1.26 million records into WL.com - feasible?
« Reply #9 on: February 01, 2022, 02:21:40 PM »
Iím beginning to understand what Jim Lovell felt like in Apollo 13.  :grin:

Iím holding back until I hear from Davis support or perhaps those here who have successfully imported and what precautions they took.

Offline Ray Proudfoot

  • Forecaster
  • *****
  • Posts: 446
    • Cheadle Hulme Weather
Re: Importing 13 years / 1.26 million records into WL.com - feasible?
« Reply #10 on: February 02, 2022, 08:32:14 AM »
I have to say Iím astonished at the lack of response to this post. It looks to me as though the number of people who have bought the WLL havenít imported any data. Maybe they just have the basic subscription which doesnít allow for historical data.

Or maybe theyíre biding their time because the import process looks fraught with danger especially when you consider it is recommended itís done in a single pass that may involve hundreds of thousands of records or even more than a million like me.

Unless you have reasonable IT skills and can import into a spreadsheet or database how do you check for dirty data? Scrolling though a huge text file is impossible.

And still no response from Davis so even if I bought the WLL and paid for a Pro subscription I feel I would be left high and dry should I encounter a problem when importing the data. On that basis Davis is failing us big time. Iím holding off until such time as the import process is radically overhauled to make it far more user friendly and multiple passes are allowed.

Offline Ray Proudfoot

  • Forecaster
  • *****
  • Posts: 446
    • Cheadle Hulme Weather
Re: Importing 13 years / 1.26 million records into WL.com - feasible?
« Reply #11 on: February 05, 2022, 06:04:35 AM »
Still no reply from Davis Support. The name seems like an oxymoron. Do they bother with customer enquiries from outside the North American continent?

Offline doubleohwhatever

  • Senior Contributor
  • ****
  • Posts: 247
Re: Importing 13 years / 1.26 million records into WL.com - feasible?
« Reply #12 on: February 05, 2022, 03:17:56 PM »
There has been a lot of consolidation in the consumer weather instruments sector over the past few years and Davis is included. They were bought by AEM.

Unfortunately, from what I've seen, a common outcome for most the these companies (Davis, Rainwise, Bloomsky, etc) after acquisition is a reduction in support staff. There also tends to be a "changing of the guard" in regards to company leadership and longtime employees that results in the remaining knowledgeable employees having a lot on their plates.

As for your particular problem, would it be possible to create a second weatherlink.com account to test the import and see what happens?


Offline Ray Proudfoot

  • Forecaster
  • *****
  • Posts: 446
    • Cheadle Hulme Weather
Re: Importing 13 years / 1.26 million records into WL.com - feasible?
« Reply #13 on: February 05, 2022, 05:31:40 PM »
After much thought I have decided not to purchase the WLL or subscribe to the service. The lack of a response from Davis is only one small part of the reason.

My priority is my website which is available free to anyone in the world. Itís fed by data from Cumulus - a much loved application no longer supported but which remains stable if lacking sophistication for graphs etc.

I have added extra features to my website and I also download the dayfile.txt from Cumulus into an Access database for my own personal use. I can interrogate it in many different ways.

If I was using just the WeatherLink program designed around 30 years ago (and it shows) then WLL would be appealing.

But I feel I wouldnít use its features enough to justify the cost and effort involved.

@doubleohwhatever, itís sad what you report. Much the same as WUnderground which is a pale shadow of what it was 10 years ago.

Offline box

  • Forecaster
  • *****
  • Posts: 325
Re: Importing 13 years / 1.26 million records into WL.com - feasible?
« Reply #14 on: February 05, 2022, 06:58:28 PM »


My priority is my website which is available free to anyone in the world. Itís fed by data from Cumulus - a much loved application no longer supported but which remains stable if lacking sophistication for graphs etc.

I have added extra features to my website and I also download the dayfile.txt from Cumulus into an Access database for my own personal use. I can interrogate it in many different ways.


Perhaps you could at least upgrade Cumulus to Cumulus MX? With the webfiles etc you could get perhaps get your graphs?

Offline Ray Proudfoot

  • Forecaster
  • *****
  • Posts: 446
    • Cheadle Hulme Weather
Re: Importing 13 years / 1.26 million records into WL.com - feasible?
« Reply #15 on: February 05, 2022, 07:17:56 PM »
Perhaps you could at least upgrade Cumulus to Cumulus MX? With the webfiles etc you could get perhaps get your graphs?

Not possible as my netbook is still running Windows XP. MX requires Windows 7 or higher. That would mean buying a second-hand laptop with an OS no longer supported. Too much effort involved and it would require a lot of work to adapt my website to MX.

The 13 year-old Samsung NC10 required a new fan two years ago and the HDD failed 10 months ago. I replaced it with an SSD meaning to two mechanical things that failed have been replaced. Should get a few more years out of it hopefully.

Offline box

  • Forecaster
  • *****
  • Posts: 325
Re: Importing 13 years / 1.26 million records into WL.com - feasible?
« Reply #16 on: February 06, 2022, 05:17:12 AM »
Perhaps you could at least upgrade Cumulus to Cumulus MX? With the webfiles etc you could get perhaps get your graphs?

Not possible as my netbook is still running Windows XP. MX requires Windows 7 or higher. That would mean buying a second-hand laptop with an OS no longer supported. Too much effort involved and it would require a lot of work to adapt my website to MX.

The 13 year-old Samsung NC10 required a new fan two years ago and the HDD failed 10 months ago. I replaced it with an SSD meaning to two mechanical things that failed have been replaced. Should get a few more years out of it hopefully.
That's a Shame. I wonder if you could go down the raspberry pi route?! They are pretty inexpensive and will run Mx on windows  or Linux. Maybe too much of a faff? :roll:

Offline Ray Proudfoot

  • Forecaster
  • *****
  • Posts: 446
    • Cheadle Hulme Weather
Re: Importing 13 years / 1.26 million records into WL.com - feasible?
« Reply #17 on: February 06, 2022, 05:24:52 AM »
That's a Shame. I wonder if you could go down the raspberry pi route?! They are pretty inexpensive and will run Mx on windows  or Linux. Maybe too much of a faff? :roll:

Thatís been suggested to me on the Cumulus forum but thatís an OS Iím unfamiliar with. Rather stick to Windows and Cumulus. It has no issues and with the lid down 99% of the time the power consumption is negligible. I have solar panels too. The basics are all displayed which is what me and my visitors are concerned with. My visitor map shows people all around the world have an interest. Ukraine and Laos being the more recent.

Offline Mapantz

  • Forecaster
  • *****
  • Posts: 648
    • Wareham Weather
Re: Importing 13 years / 1.26 million records into WL.com - feasible?
« Reply #18 on: February 08, 2022, 03:25:10 PM »
Ukraine and Laos being the more recent.

I guarantee that they are bots, Ray.  :lol:


Offline Ray Proudfoot

  • Forecaster
  • *****
  • Posts: 446
    • Cheadle Hulme Weather
Re: Importing 13 years / 1.26 million records into WL.com - feasible?
« Reply #19 on: February 08, 2022, 04:06:12 PM »
I guarantee that they are bots, Ray.  :lol:

I link to my website on an international flight sim forum so itís quite possible theyíre genuine visitors.

Oh, and still no reply from Davis Support after a week. If you want the perfect example of an oxymoron thatís it! Useless. I wonít be bothering with their WLL.

Offline WheatonRon

  • Forecaster
  • *****
  • Posts: 1187
    • WUnderground
Re: Importing 13 years / 1.26 million records into WL.com - feasible?
« Reply #20 on: February 10, 2022, 08:59:24 PM »
Today, I spent about 40 minutes on the phone with John in Davis support. My Weatherlink software, ver 6.0.5 went crazy and he carefully and methodically helped me fix the issueóin short, my data file got corrupted. My software was periodically sending CWOP, WU, MesoWest, etc., wind gusts of 327 mph so MesoWest appropriately called the reading ďsuspectĒ.  When I called Davis, a Thursday afternoon, I got right inóno hold time. I canít be any more pleased. I should have asked John about email support but forgot. Ray, I know you live outside the U.S. so not sure how to help. But those of us who live here in the USówe are lucky when dealing with Davis support, at least on the phone.

Keep in mind, Davis was helping me with software that I paid probably $100 or so 15 years ago and such software is no longer being updated!
« Last Edit: February 12, 2022, 11:16:02 AM by WheatonRon »
Davis VP2 with SHT31 (3 complete VP2 systemsó2 with a daytime fan and 1 that has a 24 hour fan); CWOP--CW5020, FW3075 and FW4350; WU--KILWHEAT17, KILWHEAT36 and KILWHEAT39; WeatherCloud.net; CoCoRaHS--IL-DP-132; and Weatherlink 2.0

Offline Ray Proudfoot

  • Forecaster
  • *****
  • Posts: 446
    • Cheadle Hulme Weather
Re: Importing 13 years / 1.26 million records into WL.com - feasible?
« Reply #21 on: February 11, 2022, 03:40:31 AM »
@WheatonRon, thanks for posting your experience with Davis Support. Phoning the US from England really isnít an option. They appear to be treating customers who use email very differently to those who phone them.

Still no reply. I wonít be buying the WLL unit. Apart from the import process being so restrictive and no edit facility for any bad data the unit still appears to have issues.


Offline Bashy

  • brecklandweather.com/pws2021
  • Forecaster
  • *****
  • Posts: 1412
  • brecklandweather.com/index.php
    • Breckland Weather
Re: Importing 13 years / 1.26 million records into WL.com - feasible?
« Reply #23 on: February 11, 2022, 07:44:33 AM »
@WheatonRon, thanks for posting your experience with Davis Support. Phoning the US from England really isnít an option. They appear to be treating customers who use email very differently to those who phone them.

Still no reply. I wonít be buying the WLL unit. Apart from the import process being so restrictive and no edit facility for any bad data the unit still appears to have issues.

Youre correct, emailing support is sporadic to say the least, can be days without a word.

Dont let that put you off WLL though, its not a bad unit, I'm not impressed with their Mobiize using a web browser though, the rain amount is still showing as inches when it should be in mm, they've known this since October.....
Kind regards
Bashy

Offline Ray Proudfoot

  • Forecaster
  • *****
  • Posts: 446
    • Cheadle Hulme Weather
Re: Importing 13 years / 1.26 million records into WL.com - feasible?
« Reply #24 on: February 11, 2022, 07:59:29 AM »
@Bashy, having given it considerable thought even if I managed to import 13 years of data into my account what the software offers me over what I have with Cumulus is simply not worth the money or effort.

My email to support last April on another topic was never answered so this isnít a one-off. It appears to be a policy for overseas customers. Their kit is great. Their support is non existent as far as Iím concerned.