Author Topic: bad data resolved (Read 3741 times)

txweather.org · « **on:** December 09, 2014, 06:53:48 PM »

Hello there!

Today out of the blues my meteohub started uploading bad data with a date tag of 12 31 1969...

[sandbox]$ grep '1969' uploadMH/*
uploadMH/MHtags.php:alltime_localdate|19691231180404:|:
uploadMH/MHtags.php:month1_localdate|19691231181214:|:
uploadMH/MHtags.php:year1_localdate|19691231180404:|:
uploadMH/MHtags.php:seqday1_utcdate|19700101001214 19691231001214 19691230001214 19691229001214 19691228001214 19691227001214 19691226001214 19691225001214 19691224001214 19691223001214 19691222001214 19691221001214 19691220001214 19691219001214 19691218001214 19691217001214 19691216001214 19691215001214 19691214001214 19691213001214 19691212001214 19691211001214 19691210001214 19691209001214 :|:
uploadMH/MHtags.php:seqday1_localdate|19691231181214 19691230181214 19691229181214 19691228181214 19691227181214 19691226181214 19691225181214 19691224181214 19691223181214 19691222181214 19691221181214 19691220181214 19691219181214 19691218181214 19691217181214 19691216181214 19691215181214 19691214181214 19691213181214 19691212181214 19691211181214 19691210181214 19691209181214 19691208181214 :|:
uploadMH/MHtags.php:seqhour6_utcdate|19700101001214 19691231181214 19691231121214 19691231061214 19691231001214 19691230181214 19691230121214 19691230061214 19691230001214 19691229181214 19691229121214 19691229061214 19691229001214 19691228181214 19691228121214 19691228061214 19691228001214 19691227181214 19691227121214 19691227061214 19691227001214 19691226181214 19691226121214 19691226061214 :|:
uploadMH/MHtags.php:seqhour6_localdate|19691231181214 19691231121214 19691231061214 19691231001214 19691230181214 19691230121214 19691230061214 19691230001214 19691229181214 19691229121214 19691229061214 19691229001214 19691228181214 19691228121214 19691228061214 19691228001214 19691227181214 19691227121214 19691227061214 19691227001214 19691226181214 19691226121214 19691226061214 19691226001214 :|:
uploadMH/MHtags.php:seqmonth1_localdate|19691231180404 :|:
uploadMH/all-sensors.xml:    <item sensor="date0" cat="date" unit="local">19691231180404</item>
uploadMH/all-sensors.xml:    <item sensor="date0" cat="date" unit="local">19691231181214</item>
uploadMH/all-sensors.xml:    <item sensor="date0" cat="date" unit="local">19691231180404</item>
[sandbox]$

I cant seem to get it cleared.... How can I fix this issue?

TIA!

txweather.org · « **Reply #1 on:** December 09, 2014, 07:01:30 PM »

It just got worst and now all my data is bad...

Bushman · « **Reply #2 on:** December 09, 2014, 07:34:42 PM »

Have you powered everything down for a full reset?

txweather.org · « **Reply #3 on:** December 09, 2014, 07:40:42 PM »

Quote from: Bushman on December 09, 2014, 07:34:42 PM

Have you powered everything down for a full reset?

I sure did.
The wind, temp, etc.. all is back but I am still pulling bad data for monthly and yearly.
Any ideas?

txweather.org · « **Reply #4 on:** December 09, 2014, 07:56:19 PM »

This is wierd.... Things seem to be clearing out... Monthly is reading correct. Still waiting on yearly.... I wonder what is causing this or what caused it.

txweather.org · « **Reply #5 on:** December 09, 2014, 09:17:19 PM »

So the system still stuck on: Season Total:[year1_rain0_total_in] in because it still displaying 12 31 1961 for yearly... Hopefully it clears on its own...

Bushman · « **Reply #6 on:** December 09, 2014, 09:19:25 PM »

Pretty weird for sure! I wonder if it is at the server end? Have you sent a note to Docbee?

txweather.org · « **Reply #7 on:** December 09, 2014, 10:20:11 PM »

Quote from: Bushman on December 09, 2014, 09:19:25 PM

Pretty weird for sure! I wonder if it is at the server end? Have you sent a note to Docbee?

I sent him and email and open up a thread at his forum...
I am a bit desperate as several followers have email me about bad data...

Also I just notice that past data for pressures looks screwed up as well...

http://www.txweather.org/index.php?p=61&lang=en#data-area

Bushman · « **Reply #8 on:** December 09, 2014, 11:51:30 PM »

Pressure and temp look metric?? Weirder. My MB is sending data fine.

txweather.org · « **Reply #9 on:** December 10, 2014, 12:33:11 AM »

I have no clue how and why is that happening. What gives... ?!?

vreihen · « **Reply #10 on:** December 10, 2014, 08:00:45 AM »

The 12/31/1969 date is a sure sign that you are west of Greenwich and your MB could not set it's date/time via NTP. (Unix/Linux use 1/1/1970 00:00:00 GMT as the start of time for clock operations, which is the evening hours of 12/31/1969 for those of us in North America. I'm guessing that you are in CST (GMT-6) given the 18:00 times in your logs.) Perhaps it did finally set the date/time after it started collecting data? I don't know how to clear the bad data in that case.....

txweather.org · « **Reply #11 on:** December 10, 2014, 09:15:12 AM »

Quote from: vreihen on December 10, 2014, 08:00:45 AM

The 12/31/1969 date is a sure sign that you are west of Greenwich and your MB could not set it's date/time via NTP. (Unix/Linux use 1/1/1970 00:00:00 GMT as the start of time for clock operations, which is the evening hours of 12/31/1969 for those of us in North America. I'm guessing that you are in CST (GMT-6) given the 18:00 times in your logs.) Perhaps it did finally set the date/time after it started collecting data? I don't know how to clear the bad data in that case.....

From the webui:
08:14 10.12.2014

From cli:
root@meteohub:~# date
Wed Dec 10 08:14:37 CST 2014
root@meteohub:~#

All is fine with time.
I am not sure why this is happening.

txweather.org · « **Reply #12 on:** December 10, 2014, 03:06:58 PM »

Now is recomputing data all over again...

wmr928fulleval(10.12.2014 10:25:54): recomputation of weather data finished: 9776178 records processed in 66891 seconds (146 records per second)
wmr928fulleval(10.12.2014 10:30:03): recomputation of weather data started.

nincehelser · « **Reply #13 on:** December 10, 2014, 06:53:22 PM »

1 Jan 1970 is the start of the unix epoch. 31 Dec 1969 would indicate you are west of the Prime Meridian.

Most likely this is due to some bug or other glitch. The question is if the time (stored as a number of seconds since the epoch started on 1 Jan 1970), or if the routine displaying the time converted to human form has gone nutty.

nincehelser · « **Reply #14 on:** December 10, 2014, 06:56:06 PM »

I guess another question would be if the meteohub is getting the time from your console or is it using its own clock?

txweather.org · « **Reply #15 on:** December 10, 2014, 08:23:23 PM »

Quote from: nincehelser on December 10, 2014, 06:56:06 PM

I guess another question would be if the meteohub is getting the time from your console or is it using its own clock?

It's getting the time from ntp and the time is fine.
The time format fixed it self automagically

Now if you go here: http://www.txweather.org/index.php?p=61&lang=en#data-area

You can see that some format is imperial and some is metric.... I have no freaking clue what is causing this...

nincehelser · « **Reply #16 on:** December 10, 2014, 08:53:58 PM »

Quote from: xcom on December 10, 2014, 08:23:23 PM

It's getting the time from ntp and the time is fine.
The time format fixed it self automagically

That's a possibility. I don't know about meteobridge in particular, but some routers do go to Jan 1, 1970 when restarted if they don't have a battery-backed real-time clock or other saved time information. They stay that way until they receive valid information from an NTP server.

I'm not sure what would be causing the other problems, though.

docbee · « **Reply #17 on:** December 11, 2014, 02:32:00 PM »

xcom is using a Meteohub, not a Meteobridge. So no router platform, it is a RPI.
The RPI also does not have a buffered RTC afaik, so when Internet connection is
weak (or DNS fails) and NTP requests on startup fail, it might set system time to epoch time 0.

txweather.org · « **Reply #18 on:** December 11, 2014, 03:50:01 PM »

Let me also add that the metric issues was a broken coded that some "how" got inserted in my template.... It was not meteohub.
The garbage time seem to have been caused by network outage. Meteohub could not recover from the network failure and cause the garbage in time.
After about 30hrs of re-computation, meteohub manage to fix the garbage time on its own.

News: