Topic: WU delivers incomplete yearly CSV data

[wvdkuil]

Affected scripts:
1. PHP Wunderground Station History Script from  Jim McMurry as used stand-alone and in Saratoga- and Leuven-template
2. WSreports Leuven-script

The link to retrieve your stations yearly data returns a CSV file which ends at September 12, 2016
https://www.wunderground.com/weatherstation/WXDailyHistory.asp?ID=xxxxxxxxx&month=01&day=01&year=2016&format=1&graphspan=year
Same for the monthly data link.

EDIT: At 2016-09-15 12:00 CEST the link returns only Sept7-15 data.

WU is constantly changing it data structures and website, so lets wait a few days to see if the data will be complete again.

Wim

[Jáchym]

Wim, its working fine for me.

Eg: my station

https://www.wunderground.com/weatherstation/WXDailyHistory.asp?ID=IJIHOMOR28&month=9&year=2016&day=13&format=1
https://www.wunderground.com/weatherstation/WXDailyHistory.asp?ID=IJIHOMOR28&month=9&year=2016&day=14&format=1&graphspan=year

[wvdkuil]

Wim, its working fine for me.

Eg: my station

https://www.wunderground.com/weatherstation/WXDailyHistory.asp?ID=IJIHOMOR28&month=9&year=2016&day=13&format=1
https://www.wunderground.com/weatherstation/WXDailyHistory.asp?ID=IJIHOMOR28&month=9&year=2016&day=14&format=1&graphspan=year

When i use your second link only a few days in September  (7 - 15)  are returned

Code: [Select]

Date,TemperatureHighC,TemperatureAvgC,TemperatureLowC,DewpointHighC,DewpointAvgC,DewpointLowC,HumidityHigh,HumidityAvg,HumidityLow,PressureMaxhPa,PressureMinhPa,WindSpeedMaxKMH,WindSpeedAvgKMH,GustSpeedMaxKMH,PrecipitationSumCM
2016-9-7,30.6,21.4,14.5,22.7,18.6,14.3,99,86,60,1026,1020,12,3,16,0.00
2016-9-8,30.2,22.1,14.7,21.7,17.9,14.5,99,80,53,1020,1015,26,4,26,0.00
2016-9-9,32.9,23.1,15.1,22.9,19.0,14.9,99,81,53,1017,1015,15,3,15,0.00
2016-9-10,33.2,23.8,16.4,22.8,19.2,16.2,99,79,49,1018,1015,15,3,16,0.00
2016-9-11,32.6,23.7,16.0,22.3,18.8,15.8,99,77,48,1020,1017,16,4,19,0.00
2016-9-12,35.1,24.9,16.3,22.4,18.9,16.1,99,73,46,1021,1018,12,4,12,0.00
2016-9-13,34.0,24.5,16.6,21.5,18.3,16.4,99,71,42,1019,1015,17,3,17,0.00
2016-9-14,32.7,24.2,15.7,21.5,17.8,12.8,99,72,46,1017,1015,19,4,21,0.00
2016-9-15,26.2,20.2,14.2,18.2,13.2,11.4,85,75,60,1017,1015,16,3,19,0.00 Same for all complaints I received this night, first it was all data from Jan1-Sep 12.
Now the same links return only data from Sept 7-15.
Also on my and others station dashboard, only a few days for this year
Just hope WU is working on it.
Wim

[Jáchym]

Aha, ok I see, sorry, I never used the annual summary I dont know what the original looked like. In my template I only work with the daily file. I just saw it goes to Sep 15 so I thought it was ok, but if it is supposed to show all days from that year then obviously there is a problem and not much we can do.

Just hope WU is working on it.

Yes....

[Jáchym]

And in fact I just noticed it is not just these CSV files, my station has no data since 7 Sep 2016 at all, as if they deleted all our history data from those views.

All my history data is gone before Sep 7

[PaulMy]

Affected scripts:
1. PHP Wunderground Station History Script from  Jim McMurry as used stand-alone and in Saratoga- and Leuven-template
2. WSreports Leuven-script

Thanks Wim, and Brian's HomeWeather PWS template is also affected.

Paul

[Jáchym]

Always good to have one´s own database

[Jáchym]

UPDATE:

Just to give you an idea of how WU works...

Yesterday I posted on both their Twitter and FB account about the problem. I thought there are three possible scenarios:
1. they will fix it
2. they will post something in the sense they are working on it
3. they will not log in and read my post yet

Well turns out I was wrong... there is a fourth option - they did not fix it, they did not post about it, but they did interact on those channels (they have posted several irrelevant posts and so obviously they must have noticed it, not that many people post there so its not like they would have to read dozens of new comments).

I understand it could be some more serious problem so fixing it cannot be immediately, but what they do is just pure ignorance and I think the only way out of this is if people really start completely ignoring WU, not visiting it, not sending data... only if they lose users and visitors they will probably think about the fact they might need to change their attitude.

[Cutty Sark Sailor]

if people really start completely ignoring WU, not visiting it, not sending data... only if they lose users and visitors they will probably think about the fact they might need to change their attitude.

er...ah...

[erikmm]

Jachym, the majority users of WU is not us weather freaks, it is users who just visits for the the temp right here and now.
They don't care for some missing "old" data, therefor loosing visitors is non excisting

[nincehelser]

And in fact I just noticed it is not just these CSV files, my station has no data since 7 Sep 2016 at all, as if they deleted all our history data from those views.

All my history data is gone before Sep 7

Another user on the Acurite support forum has noticed the same.

[Jáchym]

Erik it is not missing "some" data.

Every station I looked at has NO DATA at all prior to Sep 7 2016. In other words, all there is now is basically one week of data....

[SpringHillWeather]

I was always afraid of something like this.

Main reason I wanted my own Database stored remotely and locally on my own servers.

I guess its a good time to verify I do have all my weather station data from birth until today stored and available.

I see on Mesowest it now says you can only get data from a limited range and there was a cutoff which you need to use API to get your data.  It wouldn't surprise that getting your data for free will be limited, fee based or non-existent.   I guess we can't rely on the weather services to store our data and make it available. 

I just don't understand that if they do this then people won't upload to there service and then they don't have data,  so I'm missing something here.

[SLOweather]

Well, we'll see if this is a temporary anomaly or not.If so, it will be the reason I need to stop messing with WU. The only reason I kept it up was for the archive, and as a place to send people that wanted historic data from my station.

I dunno how they've kept it up for so long. They now say >200,000 stations. If every station reported 1x every 10 minutes, that's 20,000 stations a minute or 333 a second.

And, at 10 minutes, that's 144 records per day per station, or 28.8 million records per day, or 10.5 billion records per year.

(I know, a lot of those 200,000 stations are no longer reporting. I just checked the Dundermap for SLO, and, many moons ago, I became KCASANLU1. Now there are over 60 in this town. A lot of the newer, apparently Bloomsky stations as well as others, have been off-line for days or weeks or months.)

[Jáchym]

And this is why I created meteotemplate...

[SpringHillWeather]

Yes hopefully its temporary but with the Wundermap changes, Goldstar mystery and now the missing data... hmmm I'm not sure I have much more faith.   

I wonder who's steering the ship there?

And Jachym, yes one click now and your template saves each years data to a file.

Thanks for that again!

[Jáchym]

Just checked the FB WU page. I posted again urging them to at least give some sort of announcement about what is going on.

Instead, they posted 3 new posts about some uninteresting stuff.

Ive had enough, tomorrow I am going to put an explicit note in the template wiki discouraging people from using WU and only use it for the db update if absolutely necessary. And I am further thinking about removing all blocks and plugins that use it.

[PaulMy]

Ive had enough, tomorrow I am going to put an explicit note in the template wiki discouraging people from using WU and only use it for the db update if absolutely necessary.

Jachym, you can do that if you want, but I wouldn't... and just let users have their own experience and make their own decision, unless asked!

And I am further thinking about removing all blocks and plugins that use it.

Again it's your template so do that if you want, but it would be a disservice to those that use it, like it, and put up with it.

If someone was to ask me about WU, or any template and script I use, I would tell them my experiences and likes/dislikes but would not post those feelings unless asked - I could use BloomSky as a similar experience.  It suits some, but not necessarily all.

Paul

[Jáchym]

I will probably leave the blocks and plugins as they are, but I will explicitly discourage people from using WU at least for the updates and data import.

And this is not because "I dont like WU". It is simply because as you can see, they are EXTREMELY UNRELIABLE and also inconsistent ẃhich makes it a very bad choice for this purpose. I have seen several users whose WU data contain complete nonsense, where the data imported from WU used completely different units than ones mentoined in that file are used, people having gaps in their db when using WU for updates etc.

It is a possibility, but I would only use it if really absolutely nothing else is possible, and it is not just because I dont like it, but because the results are then not good.

[SpringHillWeather]

I think there is an underlining problem that many weather stations depend on WU to view their station and data and have no other means. 

What many failed to realize who have purchased PWS's,  is that there is no obligation from WU to to provide you with any service.  So you're stuck if they are down. 

From WU's point of view all of a sudden there are Netamo's, BloomSKy, AW's, Accrurite and others that are sending them data and this could be another problem to support all of it. 

[Mtn_man]

If you go to the WU page, and look at a weather station, and choose the monthly graph, it shows data only from Sept 7 through today.  However if you choose the daily or weekly view and choose any day prior to the 7th, it displays the data for that day.

Even if the daily or weekly graph or table is correct, the high/low table at the top does not display any values or a low temp of 9999­°F.

Appears  they took careful aim at one of their feet and pulled the trigger.

[Jáchym]

I would completely understand they might be experiencing some problems. Thats fine, no-one and nothing is perfect. What annoys me though is that absolute lack of communication.

What I also dont understand is why they have a FB page. FB is primarily a "social network" meant for communication. Yet they completely ignore everyone's comments. THere are other people asking various questions, they dont reply to a single one.

It is sort of hard to understand for me from the point of a developer. I reply to every single email I get (and people here can confirm it) and even if I was unable to solve the problem, I would look into it and let the person know what is going on - and unlike WU, I dont do this as a commercial service, in which case it is even more absurd.

[erikmm]

Jachym, write IBM who owns WU, maybe they pulled the plug.
And again, WU never promised anything.

But you are right in one thing, for the future one have to keep data at home in a DB.

Maybe the are bad to communicate, but they wouldn't be the first or last, and maybe tomorrow all data are back, who knows, I hope

But not to kept data at home, is my own and others own mistake, and at the moment I just say: It's only data and my life goes on....

[PaulMy]

But not to kept data at home, is my own and others own mistake,

I now have the "alldata' for Meteotemplate in a MySQL database but for years I have also had my dayfile in a MySQL and now also started to have realtime.txt in MySQL DB.  And I think most software like Cumulus has ongoing log files that is saved on the computer (in my case 10 min interval) and I upload these monthly files as well.  So I don't rely on WU to have my data, but it is nice to have them do it and then we can use those nice scripts like WU History, WU Graphs, PWS HomeWeather.

Paul

[Jáchym]

It almost seems to me as if they were trying to provoke people.

There are new negative comments and questions on their FB page from other users and instead, the only thing they did is posting about drinking beer at Oktoberfest in Germany.... if that is what interests them and what they do in their offices then no wonder things are the way they are