WXforum.net
Weather Software => Meteohub/Meteobridge => Topic started by: woolfg on May 07, 2019, 10:06:03 AM
-
Hi
I would like to import my Cumulus historical data going back10 years or so into my Meteobridge SQL database
I have a Cumulus dayfile.txt but I dont know how to write the SQL to get the data into the database
To see the format of the dayfile.txt I have attached a file that shows what the columns are and also a sample of the file.
I hope this would update my Almanac Data on my website
Any help would be much appreciated
Kind Regards
-
If you have meteobridge pro all you have to do is dump your cumulus data in to the import folder.... The kicker is going to be.... What does meteobridge pro expects the filename to be in the import folder.
For Meteohub I figure out that the filename had to be the date the file was generaed... IE: If I was importing raw data from 07/07/214 than the filename had to be 201407 just liek the raw folder in meteohub.
Edit: Heads up... I am currently importing 6yrs and one year alone took me a full day.
-
Hi
Thanks for the reply but I am confused
Cumulus only produces data files for each month - I dont seem to have any daily files in my data folder at all
Regards
Graham
-
Hi
Thanks for the reply but I am confused
Cumulus only produces data files for each month - I dont seem to have any daily files in my data folder at all
Regards
Graham
Ill post the instructions for you.
What version of Meteobridge Hardware you have?
-
Hi
Thanks for the reply but I am confused
Cumulus only produces data files for each month - I dont seem to have any daily files in my data folder at all
Regards
Graham
Ill post the instructions for you.
What version of Meteobridge Hardware you have?
Hi
Thats kind of you - thanks. I have a Meteobridge Pro Red
Kind Regards
Graham
-
grahm open your data share folder of your meteobridge pro and go to the folder "import" drop your cumulas data in there and go to the history tab of your meteobridge. Go to import and see if your see your data.
-
Graham, have you gotten any data from Cumulus to import into your Pro? I've been trying a handmade Cumulus month log with my Nano SD to no avail. All my data is in a custom database, so trying to convert it to something Meteobridge will understand.
-
grahm open your data share folder of your meteobridge pro and go to the folder "import" drop your cumulas data in there and go to the history tab of your meteobridge. Go to import and see if your see your data.
Hi
Yes I do see my data - I have tried all different types of the files in my Cumulus data folder, the dayfile.txt, the monthly data file and even the NOAA monthly report files. I can see all the files , I click on the import button and then get errors as shown in the attached screenshot.
Kind Regards
Graham
-
Graham, have you gotten any data from Cumulus to import into your Pro? I've been trying a handmade Cumulus month log with my Nano SD to no avail. All my data is in a custom database, so trying to convert it to something Meteobridge will understand.
Hi Lightmaster
No I havent - all my data is in flat files - I have just posted a reply to txweather which has more info.
Kind Regards
Graham
-
Graham, have you gotten any data from Cumulus to import into your Pro? I've been trying a handmade Cumulus month log with my Nano SD to no avail. All my data is in a custom database, so trying to convert it to something Meteobridge will understand.
Hi Lightmaster
No I havent - all my data is in flat files - I have just posted a reply to txweather which has more info.
Kind Regards
Graham
Same error I get. I've tried changing the time stamp on the first line to every kind I can think of and it still gives an error about the first line's timestamp.
-
Timestamp? Is this a Euro vs. NA format issue? I've seen lots of systems fail due to incorrect time formats.
-
Timestamp? Is this a Euro vs. NA format issue? I've seen lots of systems fail due to incorrect time formats.
I've tried every combination of year month day hour minute I can think of and also tried adding seconds. Even tried the exact same style of timestamp that the NANO uses in the system log
-
Timestamp? Is this a Euro vs. NA format issue? I've seen lots of systems fail due to incorrect time formats.
I've tried every combination of year month day hour minute I can think of and also tried adding seconds. Even tried the exact same style of timestamp that the NANO uses in the system log
The only other solution I can think of is for some clever person to write a SQL query that will read a Cumulus data file and write it directly to the database that is set up in the Meteobridge settings. I think the monthly data file would be best as that has all the high/low data for temperature rainfall sunshine etc
Regards
Graham
-
Meteobridge does not take daily data, it expects data with a timestamp like "2018-03-23 23:12:03"
Cumulus data is expected to look like this:
INSERT INTO `Monthly` (`LogDateTime`, `Temp`, `Humidity`, `Dewpoint`, `Windspeed`, `Windgust`, `Windbearing`, `RainRate`, `TodayRainSoFar`, `Pressure`, `Raincounter`, `InsideTemp`, `InsideHumidity`, `LatestWindGust`, `WindChill`, `HeatIndex`, `UVindex`, `SolarRad`, `Evapotrans`, `AnnualEvapTran`, `ApparentTemp`, `MaxSolarRad`, `HrsSunShine`, `CurrWindBearing`, `RG11rain`, `RainSinceMidnight`, `WindbearingSym`, `CurrWindBearingSym`) VALUES
('2018-11-30 22:45:00', '4.3', '86', '2.1', '4.0', '8.0', '223', '0.0', '1.2', '1001.66', '553.0', '21.2', '42', '2.0', '2.7', '4.3', '0.0', '0.0', '0.6', '560.4', '1.4', '0.0', '2.1', '213', '0.0', '1.2', 'SW', 'SSW'),
('2018-11-30 22:40:00', '4.3', '86', '2.1', '5.0', '12.0', '227', '0.0', '1.2', '1001.73', '553.0', '21.3', '42', '5.0', '2.3', '4.3', '0.0', '0.0', '0.6', '560.4', '1.1', '0.0', '2.1', '230', '0.0', '1.2', 'SW', 'SW'),
('2018-11-30 22:35:00', '4.3', '86', '2.1', '5.0', '12.0', '224', '0.0', '1.2', '1001.66', '553.0', '21.3', '42', '4.0', '2.3', '4.3', '0.0', '0.0', '0.6', '560.4', '1.1', '0.0', '2.1', '217', '0.0', '1.2', 'SW', 'SW'),
('2018-11-30 22:30:00', '4.3', '86', '2.1', '5.0', '12.0', '221', '0.0', '1.2', '1001.52', '553.0', '21.4', '42', '9.0', '2.3', '4.3', '0.0', '0.0', '0.6', '560.4', '1.1', '0.0', '2.1', '244', '0.0', '1.2', 'SW', 'WSW'),
('2018-11-30 22:25:00', '4.2', '86', '2.1', '5.0', '12.0', '218', '0.0', '1.2', '1001.56', '553.0', '21.5', '42', '3.0', '2.2', '4.2', '0.0', '0.0', '0.6', '560.4', '1.0', '0.0', '2.1', '250', '0.0', '1.2', 'SW', 'WSW'),
('2018-11-30 22:20:00', '4.1', '87', '2.1', '5.0', '12.0', '213', '0.0', '1.2', '1001.59', '553.0', '21.6', '42', '6.0', '2.1', '4.1', '0.0', '0.0', '0.6', '560.4', '0.9', '0.0', '2.1', '168', '0.0', '1.2', 'SSW', 'SSE'),
('2018-11-30 22:15:00', '4.1', '87', '2.1', '5.0', '12.0', '217', '0.0', '1.2', '1001.49', '553.0', '21.7', '41', '6.0', '2.1', '4.1', '0.0', '0.0', '0.6', '560.4', '0.9', '0.0', '2.1', '179', '0.0', '1.2', 'SW', 'S'),
('2018-11-30 22:10:00', '4.1', '86', '2.0', '4.0', '12.0', '218', '0.0', '1.2', '1001.52', '553.0', '21.7', '41', '2.0', '2.6', '4.1', '0.0', '0.0', '0.6', '560.4', '1.2', '0.0', '2.1', '248', '0.0', '1.2', 'SW', 'WSW'),
or this
"2016-04-20 23:15:00","6.6","89","4.9","0.0","1.0","254","0.0","0.0","1027.40","397.4","25.2","41","1.0","6.6","6.6","0.0","0.0","0.0","0.0","5.5","0.0","0.0","257","0.0","0.0","WSW","WSW"
"2016-04-20 23:20:00","6.4","89","4.8","0.0","1.0","256","0.0","0.0","1027.43","397.4","25.2","41","1.0","6.4","6.4","0.0","0.0","0.0","0.0","5.3","0.0","0.0","255","0.0","0.0","WSW","WSW"
"2016-04-20 23:25:00","6.3","89","4.7","0.0","1.0","256","0.0","0.0","1027.53","397.4","25.2","41","0.0","6.3","6.3","0.0","0.0","0.0","0.0","5.1","0.0","0.0","256","0.0","0.0","WSW","WSW"
"2016-04-20 23:30:00","6.3","89","4.6","0.0","2.0","259","0.0","0.0","1027.53","397.4","25.2","41","0.0","6.3","6.3","0.0","0.0","0.0","0.0","5.1","0.0","0.0","264","0.0","0.0","W","W"
"2016-04-20 23:35:00","6.2","89","4.5","1.0","2.0","264","0.0","0.0","1027.70","397.4","25.2","41","0.0","6.2","6.2","0.0","0.0","0.0","0.0","4.7","0.0","0.0","273","0.0","0.0","W","W"
"2016-04-20 23:40:00","6.2","89","4.5","1.0","2.0","263","0.0","0.0","1027.74","397.4","25.2","41","2.0","6.2","6.2","0.0","0.0","0.0","0.0","4.6","0.0","0.0","270","0.0","0.0","W","W"
"2016-04-20 23:45:00","6.2","89","4.5","0.0","2.0","260","0.0","0.0","1027.70","397.4","25.2","41","0.0","6.2","6.2","0.0","0.0","0.0","0.0","4.9","0.0","0.0","270","0.0","0.0","W","W"
"2016-04-20 23:50:00","6.2","89","4.5","0.0","2.0","270","0.0","0.0","1027.63","397.4","25.1","41","0.0","6.2","6.2","0.0","0.0","0.0","0.0","4.9","0.0","0.0","270","0.0","0.0","W","W"
"2016-04-20 23:55:00","6.1","89","4.4","0.0","1.0","270","0.0","0.0","1027.67","397.4","25.2","41","0.0","6.1","6.1","0.0","0.0","0.0","0.0","4.9","0.0","0.0","270","0.0","0.0","W","W"
"2016-04-21 00:00:00","6.1","89","4.4","0.0","1.0","270","0.0","0.0","1027.74","397.4","25.1","41","0.0","6.1","6.1","0.0","0.0","0.0","0.0","4.9","0.0","0.0","270","0.0","0.0","W","W"
"2016-04-21 00:05:00","6.1","89","4.4","0.0","1.0","270","0.0","0.0","1027.84","397.4","25.1","40","0.0","6.1","6.1","0.0","0.0","0.0","0.0","4.8","0.0","0.0","270","0.0","0.0","W","W"
"2016-04-21 00:10:00","6.1","89","4.4","0.0","1.0","270","0.0","0.0","1027.87","397.4","25.1","40","0.0","6.1","6.1","0.0","0.0","0.0","0.0","4.8","0.0","0.0","270","0.0","0.0","W","W"
"2016-04-21 00:15:00","5.9","88","4.1","0.0","1.0","270","0.0","0.0","1027.87","397.4","25.1","40","0.0","5.9","5.9","0.0","0.0","0.0","0.0","4.6","0.0","0.0","269","0.0","0.0","W","W"
"2016-04-21 00:20:00","5.8","88","4.0","0.0","0.0","0","0.0","0.0","1027.87","397.4","25.0","40","0.0","5.8","5.8","0.0","0.0","0.0","0.0","4.5","0.0","0.0","270","0.0","0.0","N","W"
-
Hi Docbee
Thanks for the reply.
It looks like a lot of work to convert the Cumulus data files into a format that the Weather34 MySQL database would understand which is a shame.
If I reinstated my Envoy Data logger and Cumulus software and then connected it to my MeteoBridge Pro would my historical data get uploaded that way
Regards
Graham
-
Hi Docbee
Thanks for the reply.
It looks like a lot of work to convert the Cumulus data files into a format that the Weather34 MySQL database would understand which is a shame.
If I reinstated my Envoy Data logger and Cumulus software and then connected it to my MeteoBridge Pro would my historical data get uploaded that way
Regards
Graham
I'm assuming you meant the Meteobridge SQLite database... As far as I can tell, only difference seems be be the data time format. Gonna try to import a month of mine in a bit when I get a chance.
Meteobridge does not take daily data, it expects data with a timestamp like "2018-03-23 23:12:03"
Cumulus data is expected to look like this:
INSERT INTO `Monthly` (`LogDateTime`, `Temp`, `Humidity`, `Dewpoint`, `Windspeed`, `Windgust`, `Windbearing`, `RainRate`, `TodayRainSoFar`, `Pressure`, `Raincounter`, `InsideTemp`, `InsideHumidity`, `LatestWindGust`, `WindChill`, `HeatIndex`, `UVindex`, `SolarRad`, `Evapotrans`, `AnnualEvapTran`, `ApparentTemp`, `MaxSolarRad`, `HrsSunShine`, `CurrWindBearing`, `RG11rain`, `RainSinceMidnight`, `WindbearingSym`, `CurrWindBearingSym`) VALUES
('2018-11-30 22:45:00', '4.3', '86', '2.1', '4.0', '8.0', '223', '0.0', '1.2', '1001.66', '553.0', '21.2', '42', '2.0', '2.7', '4.3', '0.0', '0.0', '0.6', '560.4', '1.4', '0.0', '2.1', '213', '0.0', '1.2', 'SW', 'SSW'),
('2018-11-30 22:40:00', '4.3', '86', '2.1', '5.0', '12.0', '227', '0.0', '1.2', '1001.73', '553.0', '21.3', '42', '5.0', '2.3', '4.3', '0.0', '0.0', '0.6', '560.4', '1.1', '0.0', '2.1', '230', '0.0', '1.2', 'SW', 'SW'),
('2018-11-30 22:35:00', '4.3', '86', '2.1', '5.0', '12.0', '224', '0.0', '1.2', '1001.66', '553.0', '21.3', '42', '4.0', '2.3', '4.3', '0.0', '0.0', '0.6', '560.4', '1.1', '0.0', '2.1', '217', '0.0', '1.2', 'SW', 'SW'),
('2018-11-30 22:30:00', '4.3', '86', '2.1', '5.0', '12.0', '221', '0.0', '1.2', '1001.52', '553.0', '21.4', '42', '9.0', '2.3', '4.3', '0.0', '0.0', '0.6', '560.4', '1.1', '0.0', '2.1', '244', '0.0', '1.2', 'SW', 'WSW'),
('2018-11-30 22:25:00', '4.2', '86', '2.1', '5.0', '12.0', '218', '0.0', '1.2', '1001.56', '553.0', '21.5', '42', '3.0', '2.2', '4.2', '0.0', '0.0', '0.6', '560.4', '1.0', '0.0', '2.1', '250', '0.0', '1.2', 'SW', 'WSW'),
('2018-11-30 22:20:00', '4.1', '87', '2.1', '5.0', '12.0', '213', '0.0', '1.2', '1001.59', '553.0', '21.6', '42', '6.0', '2.1', '4.1', '0.0', '0.0', '0.6', '560.4', '0.9', '0.0', '2.1', '168', '0.0', '1.2', 'SSW', 'SSE'),
('2018-11-30 22:15:00', '4.1', '87', '2.1', '5.0', '12.0', '217', '0.0', '1.2', '1001.49', '553.0', '21.7', '41', '6.0', '2.1', '4.1', '0.0', '0.0', '0.6', '560.4', '0.9', '0.0', '2.1', '179', '0.0', '1.2', 'SW', 'S'),
('2018-11-30 22:10:00', '4.1', '86', '2.0', '4.0', '12.0', '218', '0.0', '1.2', '1001.52', '553.0', '21.7', '41', '2.0', '2.6', '4.1', '0.0', '0.0', '0.6', '560.4', '1.2', '0.0', '2.1', '248', '0.0', '1.2', 'SW', 'WSW'),
or this
"2016-04-20 23:15:00","6.6","89","4.9","0.0","1.0","254","0.0","0.0","1027.40","397.4","25.2","41","1.0","6.6","6.6","0.0","0.0","0.0","0.0","5.5","0.0","0.0","257","0.0","0.0","WSW","WSW"
"2016-04-20 23:20:00","6.4","89","4.8","0.0","1.0","256","0.0","0.0","1027.43","397.4","25.2","41","1.0","6.4","6.4","0.0","0.0","0.0","0.0","5.3","0.0","0.0","255","0.0","0.0","WSW","WSW"
"2016-04-20 23:25:00","6.3","89","4.7","0.0","1.0","256","0.0","0.0","1027.53","397.4","25.2","41","0.0","6.3","6.3","0.0","0.0","0.0","0.0","5.1","0.0","0.0","256","0.0","0.0","WSW","WSW"
"2016-04-20 23:30:00","6.3","89","4.6","0.0","2.0","259","0.0","0.0","1027.53","397.4","25.2","41","0.0","6.3","6.3","0.0","0.0","0.0","0.0","5.1","0.0","0.0","264","0.0","0.0","W","W"
"2016-04-20 23:35:00","6.2","89","4.5","1.0","2.0","264","0.0","0.0","1027.70","397.4","25.2","41","0.0","6.2","6.2","0.0","0.0","0.0","0.0","4.7","0.0","0.0","273","0.0","0.0","W","W"
"2016-04-20 23:40:00","6.2","89","4.5","1.0","2.0","263","0.0","0.0","1027.74","397.4","25.2","41","2.0","6.2","6.2","0.0","0.0","0.0","0.0","4.6","0.0","0.0","270","0.0","0.0","W","W"
"2016-04-20 23:45:00","6.2","89","4.5","0.0","2.0","260","0.0","0.0","1027.70","397.4","25.2","41","0.0","6.2","6.2","0.0","0.0","0.0","0.0","4.9","0.0","0.0","270","0.0","0.0","W","W"
"2016-04-20 23:50:00","6.2","89","4.5","0.0","2.0","270","0.0","0.0","1027.63","397.4","25.1","41","0.0","6.2","6.2","0.0","0.0","0.0","0.0","4.9","0.0","0.0","270","0.0","0.0","W","W"
"2016-04-20 23:55:00","6.1","89","4.4","0.0","1.0","270","0.0","0.0","1027.67","397.4","25.2","41","0.0","6.1","6.1","0.0","0.0","0.0","0.0","4.9","0.0","0.0","270","0.0","0.0","W","W"
"2016-04-21 00:00:00","6.1","89","4.4","0.0","1.0","270","0.0","0.0","1027.74","397.4","25.1","41","0.0","6.1","6.1","0.0","0.0","0.0","0.0","4.9","0.0","0.0","270","0.0","0.0","W","W"
"2016-04-21 00:05:00","6.1","89","4.4","0.0","1.0","270","0.0","0.0","1027.84","397.4","25.1","40","0.0","6.1","6.1","0.0","0.0","0.0","0.0","4.8","0.0","0.0","270","0.0","0.0","W","W"
"2016-04-21 00:10:00","6.1","89","4.4","0.0","1.0","270","0.0","0.0","1027.87","397.4","25.1","40","0.0","6.1","6.1","0.0","0.0","0.0","0.0","4.8","0.0","0.0","270","0.0","0.0","W","W"
"2016-04-21 00:15:00","5.9","88","4.1","0.0","1.0","270","0.0","0.0","1027.87","397.4","25.1","40","0.0","5.9","5.9","0.0","0.0","0.0","0.0","4.6","0.0","0.0","269","0.0","0.0","W","W"
"2016-04-21 00:20:00","5.8","88","4.0","0.0","0.0","0","0.0","0.0","1027.87","397.4","25.0","40","0.0","5.8","5.8","0.0","0.0","0.0","0.0","4.5","0.0","0.0","270","0.0","0.0","N","W"
Does data have to be in metric, or is MB capable of recognizing the difference? Obviously if temp is 90, its not Celcius, lol. If it has to be in metric, not the end of the world for me, just gotta export it from my database again but with metric conversions in it.
-
Combining the date and time columns as YYYY-MM-DD hh:mm:ss worked. File doesn't have seconds, so I just added 00 as the seconds.
-
Combining the date and time columns as YYYY-MM-DD hh:mm:ss worked. File does have seconds, so I just added 00 as the seconds.
Hi Lightmaster
OK thats encouraging - which cumulus file did you use . I have filenames like FEB19log.txt. Also how did you convert the date/time - is there a quick way of doing it.
Regards
Graham
-
I just added a section about data imports to the wiki. That might help as well.
https://www.meteobridge.com/wiki/index.php/PRO_Tutorial#Data_Import
-
I just added a section about data imports to the wiki. That might help as well.
https://www.meteobridge.com/wiki/index.php/PRO_Tutorial#Data_Import
Documentation on the import capabilities looks great. Might wanna add a link in the Nano SD and Pro product pages to direct people to that documentation.
-
I just added a section about data imports to the wiki. That might help as well.
https://www.meteobridge.com/wiki/index.php/PRO_Tutorial#Data_Import
Hi Docbee
Thanks for that - unfortunately my cumulus data files have hundreds of lines in them and I can see no easy way of converting them unless I do it manually. Mine look like this
01/09/18,09:00,16.7,90,15.1,0.2,2.6,99,0.0,0.0,1025.18,122.4,20.7,67,0.0,16.7,16.7,11.0,218,2.03,544.86,18.3,317,0.4,0,0.0,0.0
01/09/18,09:05,16.8,90,15.1,0.5,3.5,68,0.0,0.0,1025.25,122.4,20.7,68,3.5,16.8,16.8,11.0,217,0.10,544.96,18.2,331,0.4,67,0.0,0.0
01/09/18,09:10,16.9,89,15.1,0.9,3.5,68,0.0,0.0,1025.28,122.4,20.8,69,2.6,16.9,16.9,10.9,206,0.10,544.96,18.3,345,0.4,67,0.0,0.0
01/09/18,09:15,17.1,89,15.2,0.9,5.2,68,0.0,0.0,1025.35,122.4,20.8,69,5.2,17.1,17.1,10.9,152,0.10,544.96,18.4,359,0.4,67,0.0,0.0
01/09/18,09:30,17.7,88,15.7,0.9,3.5,73,0.0,0.0,1025.52,122.4,20.9,70,0.9,17.7,17.7,10.8,167,0.10,544.96,19.2,401,0.4,48,0.0,0.0
01/09/18,09:45,18.4,87,16.2,1.3,5.2,97,0.0,0.0,1025.58,122.4,21.2,70,0.9,18.4,18.4,10.7,250,0.10,544.96,20.0,441,0.4,92,0.0,0.0
01/09/18,10:00,19.3,85,16.7,1.3,3.5,178,0.0,0.0,1025.58,122.4,21.8,69,1.7,19.3,19.3,10.6,283,0.10,544.96,21.1,479,0.4,184,0.0,0.0
01/09/18,10:15,19.6,83,16.6,1.4,3.5,158,0.0,0.0,1025.62,122.4,22.3,68,0.9,19.6,19.6,10.7,309,0.23,545.08,21.3,516,0.4,240,0.0,0.0
01/09/18,10:30,19.9,80,16.3,1.0,4.3,167,0.0,0.0,1025.62,122.4,22.6,66,0.9,19.9,19.9,10.7,327,0.23,545.08,21.7,549,0.4,108,0.0,0.0
01/09/18,10:45,20.1,79,16.3,1.2,6.1,163,0.0,0.0,1025.72,122.4,22.9,65,0.9,20.1,20.1,10.9,341,0.23,545.08,21.8,581,0.4,117,0.0,0.0
I dont have the expertise either in Excel of programming to write a utility that would ease the conversion
Regards
Graham
-
I just added a section about data imports to the wiki. That might help as well.
https://www.meteobridge.com/wiki/index.php/PRO_Tutorial#Data_Import
Hi Docbee
Thanks for that - unfortunately my cumulus data files have hundreds of lines in them and I can see no easy way of converting them unless I do it manually. Mine look like this
01/09/18,09:00,16.7,90,15.1,0.2,2.6,99,0.0,0.0,1025.18,122.4,20.7,67,0.0,16.7,16.7,11.0,218,2.03,544.86,18.3,317,0.4,0,0.0,0.0
01/09/18,09:05,16.8,90,15.1,0.5,3.5,68,0.0,0.0,1025.25,122.4,20.7,68,3.5,16.8,16.8,11.0,217,0.10,544.96,18.2,331,0.4,67,0.0,0.0
01/09/18,09:10,16.9,89,15.1,0.9,3.5,68,0.0,0.0,1025.28,122.4,20.8,69,2.6,16.9,16.9,10.9,206,0.10,544.96,18.3,345,0.4,67,0.0,0.0
01/09/18,09:15,17.1,89,15.2,0.9,5.2,68,0.0,0.0,1025.35,122.4,20.8,69,5.2,17.1,17.1,10.9,152,0.10,544.96,18.4,359,0.4,67,0.0,0.0
01/09/18,09:30,17.7,88,15.7,0.9,3.5,73,0.0,0.0,1025.52,122.4,20.9,70,0.9,17.7,17.7,10.8,167,0.10,544.96,19.2,401,0.4,48,0.0,0.0
01/09/18,09:45,18.4,87,16.2,1.3,5.2,97,0.0,0.0,1025.58,122.4,21.2,70,0.9,18.4,18.4,10.7,250,0.10,544.96,20.0,441,0.4,92,0.0,0.0
01/09/18,10:00,19.3,85,16.7,1.3,3.5,178,0.0,0.0,1025.58,122.4,21.8,69,1.7,19.3,19.3,10.6,283,0.10,544.96,21.1,479,0.4,184,0.0,0.0
01/09/18,10:15,19.6,83,16.6,1.4,3.5,158,0.0,0.0,1025.62,122.4,22.3,68,0.9,19.6,19.6,10.7,309,0.23,545.08,21.3,516,0.4,240,0.0,0.0
01/09/18,10:30,19.9,80,16.3,1.0,4.3,167,0.0,0.0,1025.62,122.4,22.6,66,0.9,19.9,19.9,10.7,327,0.23,545.08,21.7,549,0.4,108,0.0,0.0
01/09/18,10:45,20.1,79,16.3,1.2,6.1,163,0.0,0.0,1025.72,122.4,22.9,65,0.9,20.1,20.1,10.9,341,0.23,545.08,21.8,581,0.4,117,0.0,0.0
I dont have the expertise either in Excel of programming to write a utility that would ease the conversion
Regards
Graham
Yours are all in metric already, at least that I can see. Download VSCode (a coding text editor). If you open a file and then press Ctrl+Alt+down arrow, keep pressing the down arrow and you can have a cursor on every line, then using cut and paste, you can rearrange an entire file at once so it's in the right order. Typing and delete also work like this, so you can delete the extra comma and type in the 20 in 2019, 2018, etc.
Basically, edit one line and the entire file is edited
-
I just added a section about data imports to the wiki. That might help as well.
https://www.meteobridge.com/wiki/index.php/PRO_Tutorial#Data_Import
Hi Docbee
Thanks for that - unfortunately my cumulus data files have hundreds of lines in them and I can see no easy way of converting them unless I do it manually. Mine look like this
01/09/18,09:00,16.7,90,15.1,0.2,2.6,99,0.0,0.0,1025.18,122.4,20.7,67,0.0,16.7,16.7,11.0,218,2.03,544.86,18.3,317,0.4,0,0.0,0.0
01/09/18,09:05,16.8,90,15.1,0.5,3.5,68,0.0,0.0,1025.25,122.4,20.7,68,3.5,16.8,16.8,11.0,217,0.10,544.96,18.2,331,0.4,67,0.0,0.0
01/09/18,09:10,16.9,89,15.1,0.9,3.5,68,0.0,0.0,1025.28,122.4,20.8,69,2.6,16.9,16.9,10.9,206,0.10,544.96,18.3,345,0.4,67,0.0,0.0
01/09/18,09:15,17.1,89,15.2,0.9,5.2,68,0.0,0.0,1025.35,122.4,20.8,69,5.2,17.1,17.1,10.9,152,0.10,544.96,18.4,359,0.4,67,0.0,0.0
01/09/18,09:30,17.7,88,15.7,0.9,3.5,73,0.0,0.0,1025.52,122.4,20.9,70,0.9,17.7,17.7,10.8,167,0.10,544.96,19.2,401,0.4,48,0.0,0.0
01/09/18,09:45,18.4,87,16.2,1.3,5.2,97,0.0,0.0,1025.58,122.4,21.2,70,0.9,18.4,18.4,10.7,250,0.10,544.96,20.0,441,0.4,92,0.0,0.0
01/09/18,10:00,19.3,85,16.7,1.3,3.5,178,0.0,0.0,1025.58,122.4,21.8,69,1.7,19.3,19.3,10.6,283,0.10,544.96,21.1,479,0.4,184,0.0,0.0
01/09/18,10:15,19.6,83,16.6,1.4,3.5,158,0.0,0.0,1025.62,122.4,22.3,68,0.9,19.6,19.6,10.7,309,0.23,545.08,21.3,516,0.4,240,0.0,0.0
01/09/18,10:30,19.9,80,16.3,1.0,4.3,167,0.0,0.0,1025.62,122.4,22.6,66,0.9,19.9,19.9,10.7,327,0.23,545.08,21.7,549,0.4,108,0.0,0.0
01/09/18,10:45,20.1,79,16.3,1.2,6.1,163,0.0,0.0,1025.72,122.4,22.9,65,0.9,20.1,20.1,10.9,341,0.23,545.08,21.8,581,0.4,117,0.0,0.0
I dont have the expertise either in Excel of programming to write a utility that would ease the conversion
Regards
Graham
Is this a one-time thing for you? Or are you planning on regularly importing these monthly files? It is pretty simple to do in Excel (although I'd probably use AWK (GAWK for Windows). Another alternative is Notepad++ BUt if you are doing this regularly, a simple piece of code could be written.
-
I just added a section about data imports to the wiki. That might help as well.
https://www.meteobridge.com/wiki/index.php/PRO_Tutorial#Data_Import
Hi Docbee
Thanks for that - unfortunately my cumulus data files have hundreds of lines in them and I can see no easy way of converting them unless I do it manually. Mine look like this
01/09/18,09:00,16.7,90,15.1,0.2,2.6,99,0.0,0.0,1025.18,122.4,20.7,67,0.0,16.7,16.7,11.0,218,2.03,544.86,18.3,317,0.4,0,0.0,0.0
01/09/18,09:05,16.8,90,15.1,0.5,3.5,68,0.0,0.0,1025.25,122.4,20.7,68,3.5,16.8,16.8,11.0,217,0.10,544.96,18.2,331,0.4,67,0.0,0.0
01/09/18,09:10,16.9,89,15.1,0.9,3.5,68,0.0,0.0,1025.28,122.4,20.8,69,2.6,16.9,16.9,10.9,206,0.10,544.96,18.3,345,0.4,67,0.0,0.0
01/09/18,09:15,17.1,89,15.2,0.9,5.2,68,0.0,0.0,1025.35,122.4,20.8,69,5.2,17.1,17.1,10.9,152,0.10,544.96,18.4,359,0.4,67,0.0,0.0
01/09/18,09:30,17.7,88,15.7,0.9,3.5,73,0.0,0.0,1025.52,122.4,20.9,70,0.9,17.7,17.7,10.8,167,0.10,544.96,19.2,401,0.4,48,0.0,0.0
01/09/18,09:45,18.4,87,16.2,1.3,5.2,97,0.0,0.0,1025.58,122.4,21.2,70,0.9,18.4,18.4,10.7,250,0.10,544.96,20.0,441,0.4,92,0.0,0.0
01/09/18,10:00,19.3,85,16.7,1.3,3.5,178,0.0,0.0,1025.58,122.4,21.8,69,1.7,19.3,19.3,10.6,283,0.10,544.96,21.1,479,0.4,184,0.0,0.0
01/09/18,10:15,19.6,83,16.6,1.4,3.5,158,0.0,0.0,1025.62,122.4,22.3,68,0.9,19.6,19.6,10.7,309,0.23,545.08,21.3,516,0.4,240,0.0,0.0
01/09/18,10:30,19.9,80,16.3,1.0,4.3,167,0.0,0.0,1025.62,122.4,22.6,66,0.9,19.9,19.9,10.7,327,0.23,545.08,21.7,549,0.4,108,0.0,0.0
01/09/18,10:45,20.1,79,16.3,1.2,6.1,163,0.0,0.0,1025.72,122.4,22.9,65,0.9,20.1,20.1,10.9,341,0.23,545.08,21.8,581,0.4,117,0.0,0.0
I dont have the expertise either in Excel of programming to write a utility that would ease the conversion
Regards
Graham
Is this a one-time thing for you? Or are you planning on regularly importing these monthly files? It is pretty simple to do in Excel (although I'd probably use AWK (GAWK for Windows). Another alternative is Notepad++ BUt if you are doing this regularly, a simple piece of code could be written.
I am a bit lost with this whole import thing... Let me elaborate.... If I have a native Meteohub raw file.... Why do I have to convert the file to a format that the nano sd understands? This seems like something that could have been written on the import process instead of saying "you can import files from meteohub" when in reality you really cant...
I have a full year of meteohub data that I thought I was importing "fine" to the nano sd but it turns out that now my nano sd has a crap load of bad data fort hat whole year!
Here is a snippet of the meteohub raw data:
20180701000400 thb1 262 51 0153 10077 -1 10135
20180701000400 th0 273 87 0249
20180701000400 wind0 158 36 18 0273
20180701000400 rain0 0 0 1524
20180701000900 thb1 263 55 0165 10077 -1 10135
20180701000900 th0 273 87 0249
20180701000900 wind0 158 31 18 0273
20180701000900 rain0 0 0 1524
20180701001400 thb1 264 54 0163 10077 -1 10135
20180701001400 th0 273 87 0249
20180701001400 wind0 180 31 13 0273
20180701001400 rain0 0 0 1524
20180701001900 thb1 265 51 0155 10077 -1 10135
20180701001900 th0 273 87 0249
20180701001900 wind0 180 27 13 0273
20180701001900 rain0 0 0 1524
20180701002400 thb1 263 49 0147 10078 -1 10136
20180701002400 th0 272 87 0248
20180701002400 wind0 203 22 13 0272
20180701002400 rain0 0 0 1524
20180701002900 thb1 262 51 0153 10077 -1 10135
20180701002900 th0 272 87 0248
20180701002900 wind0 158 22 13 0272
20180701002900 rain0 0 0 1524
20180701003400 thb1 263 53 0160 10076 -1 10134
20180701003400 th0 272 87 0248
20180701003400 wind0 158 22 13 0272
20180701003400 rain0 0 0 1524
20180701003900 thb1 264 57 0172 10076 -1 10134
20180701003900 th0 271 87 0247
20180701003900 wind0 158 22 13 0271
20180701003900 rain0 0 0 1524
And for the year of 2018 everything that I imported is completely off temperature wise... Its saying July in Texas was 32F! LOL
To me it seem like is just not plain supported unless converted to what ever the nano expects.
-
the beteljuice needed a break from some truly astronomical math coding, so ......
Try Cumulus month log > MB import format (http://www.beteljuice.co.uk/DEMOS/convert.php)
I have no data files nor weather software, so this should be regarded as risky !
It will convert date / time format, convert units where necessary and it will create a .txt file in the format of the above posts.
No idea if it will actually 'import' or not !!!
Edit: Now proven to 'import' (Thanks Kocher)
Edit: Handles Cu files from ver 1.8.4 > 1.9.5 and mx.
Multiple file selection allowed - ie. marking a year of logs will produce ONE large 'import' file.
(Any missing fields are 0, 0.0, or "N" as appropriate)
Use 'debug' on your first run so you can easily check if any conversions have been applied correctly.
Let me know if your PC catches fire ..... :shock:
-
It works... :thumbsup:
Enjoy,
Paul
-
.... but
... will it 'import' ?? - and you are already using the requested Units Of Measure.
-
...aaaaaaaaah
and the converted data is wrong #-o
Using the same metric units"2019-05-01 00:00:00","5.7","77","1.9","12.9","36.4","1","0.0","0.0","1023.98","222.8","26.1","28","15.0","3.2","5.7","0.0","0","1.22","147.96","1.5","0","0.2","360","0.0","0.0","N","N"
Converting to Imperial units"2019-05-01 00:00:00","-14.6","77","-16.7","18.9","53.4","1","0.0","0.0","34675.94","5659.1","-3.3","28","22.0","-16.0","-14.6","0.0","0","1.22","147.96","-16.9","0","0.2","360","0.0","0.0","N","N"
Enjoy,
Paul
-
and the converted data is wrong - Using the same metric units, ... Converting to Imperial units
???? It doesn't convert to Imperial ...
It asks you what your units are and converts them to the required ones.
This:
"2019-05-01 00:00:00","5.7","77","1.9", after being told it's Imperial
becomes
"2019-05-01 00:00:00","-14.6","77","-16.7", as metric
.. or am I misunderstanding you ?
-
My Cumulus monthly source file is metric.
In the first test I selected the metric units in your utility and that seemed to convert ok to metric to the MB format. =D>
My second test was wrong as I then selected the imperial units :oops:
I don't have MB so don't know if the converted format file can be imported.
Enjoy,
Paul
-
I don't have MB so don't know if the converted format file can be imported.
Tsk Paul - I thought you had the betas of everything that's out there :-P
... Anyone who can check if it 'imports' or not ????
-
the beteljuice needed a break from some truly astronomical math coding, so ......
Try Cumulus month log > MB import format (http://www.beteljuice.co.uk/DEMOS/convert.php)
I have no data files nor weather software, so this should be regarded as risky !
It should convert date / time format, convert units where necessary and it will create a .txt file in the format of the above posts.
No idea if it will actually 'import' or not !!!
Great - I'll give it a try
Use 'debug' on your first run so you can easily check if any conversions have been applied correctly.
Let me know if your PC catches fire ..... :shock:
-
the beteljuice needed a break from some truly astronomical math coding, so ......
Try Cumulus month log > MB import format (http://www.beteljuice.co.uk/DEMOS/convert.php)
I have no data files nor weather software, so this should be regarded as risky !
It should convert date / time format, convert units where necessary and it will create a .txt file in the format of the above posts.
No idea if it will actually 'import' or not !!!
Great - I'll give it a try
Use 'debug' on your first run so you can easily check if any conversions have been applied correctly.
Let me know if your PC catches fire ..... :shock:
I'll give it a try
-
the beteljuice needed a break from some truly astronomical math coding, so ......
Try Cumulus month log > MB import format (http://www.beteljuice.co.uk/DEMOS/convert.php)
I have no data files nor weather software, so this should be regarded as risky !
It should convert date / time format, convert units where necessary and it will create a .txt file in the format of the above posts.
No idea if it will actually 'import' or not !!!
Great - I'll give it a try
Use 'debug' on your first run so you can easily check if any conversions have been applied correctly.
Let me know if your PC catches fire ..... :shock:
I'll give it a try
It doesnt work for me -it seems to fail on the pressure for some reason. I select mb as my pressure data is 1017.34 for example but it doesnt seem to like that format
-
Greetings Beteljuice
Saludos Beteljuice
First, I appreciate your passion for the games.
I have tested your utility to import cumulus data (meteobridge Pro red (http://kocher.es/cajon/mbpro.png)) and, it worked.
My source file is in a metric format with a decimal sign (,).
It becomes your page well, but it loses the decimal value; for example: 9.6 becomes: 9
To solve this, I have made a previous change of the (,) by (.)
After this, the conversion has been done perfectly on your page.
Meteohub has imported all the data, without any error message.
The only slight inconvenience that I have been able to observe is that the rainfall data, on the days that it rained in abundance, are not exact; they are undervalued. But I must clarify that in your file, these data are correct; I think it is due to a problem of meteobridge itself.
- Javier
-
FIXED:
Decimals 'lost' on format conversion and comma (,) decimal not being correctly dealt with in o/p file.
... both the same problem. the beteljuice had a slightly mistyped var name in an important place :oops:
Naughty beteljuice #-o
I have tested your utility to import cumulus data (meteobridge Pro red ) and, it worked.
Clever betejuice \:D/
Please try again .....
redone converter (same link) (http://www.beteljuice.co.uk/DEMOS/convert.php)
-
Hi beteljuice
I have tried your converter without success
I have attached a sample of my data (unfortunately I have cumulus data going back about 9 years which I would love to convert)
Can you have a look at my data and see if you can spot what is wrong
Kind regards
Graham
-
FIXED:
Decimals 'lost' on format conversion and comma (,) decimal not being correctly dealt with in o/p file.
... both the same problem. the beteljuice had a slightly mistyped var name in an important place :oops:
Naughty beteljuice #-o
I have tested your utility to import cumulus data (meteobridge Pro red ) and, it worked.
Clever betejuice \:D/
Please try again .....
redone converter (same link) (http://www.beteljuice.co.uk/DEMOS/convert.php)
Now it works perfectly.
Original file created by cumulusMX:
01/01/19;00:00;5,3;93;4,3;1,6;4,8;154;0,0;0,0;1033,39;1478,2;20,3;54;0,0;5,3;5,3;0,0;0;0,00;793,37;3,8;0;0,0;154;0,0;0,0
01/01/19;00:05;5,3;93;4,3;0,0;3,2;154;0,0;0,0;1033,36;0,0;20,3;54;0,0;5,3;5,3;0,0;0;-793,37;0,00;4,1;0;0,0;154;0,0;0,0
01/01/19;00:10;5,4;93;4,4;0,0;3,2;154;0,0;0,0;1033,42;0,0;20,3;54;0,0;5,4;5,4;0,0;0;-793,37;0,00;4,2;0;0,0;154;0,0;0,0
The same file, without any modification on my part, converted by Beteljuice:
"2019-01-01 00:00:00","5.3","93","4.3","1.6","4.8","154","0.0","0.0","1033.39","1478.2","20.3","54","0.0","5.3","5.3","0.0","0","0.00","793.37","3.8","0","0.0","154","0.0","0.0","N","SE"
"2019-01-01 00:05:00","5.3","93","4.3","0.0","3.2","154","0.0","0.0","1033.36","0.0","20.3","54","0.0","5.3","5.3","0.0","0","-793.37","0.00","4.1","0","0.0","154","0.0","0.0","N","SE"
"2019-01-01 00:10:00","5.4","93","4.4","0.0","3.2","154","0.0","0.0","1033.42","0.0","20.3","54","0.0","5.4","5.4","0.0","0","-793.37","0.00","4.2","0","0.0","154","0.0","0.0","N","SE"
You just have to put the cumulus file as it was created at the time.
Then the type of units we want are chosen.
we activate "debug" and ... Perfect!
meteobridge imports it without any problem
Betetjuice (https://www.wxforum.net/index.php?action=dlattach;attach=33258;type=avatar) is a phenomenon! =D> =D> =D>
- Javier
-
@woolfg
That line appears to be in MB import format but without the " and last two compass points missing.
Is that your source file or the converter o/p ?
The converter would NOT recognise that as Cumulus data.
My test data was the few lines you posted earlier in the thread, and that works OK ....
01/09/18,09:00,16.7,90,15.1,0.2,2.6,99,0.0,0.0,1025.18,122.4,20.7,67,0.0,16.7,16.7,11.0,218,2.03,544.86,18.3,317,0.4,0,0.0,0.0
01/09/18,09:05,16.8,90,15.1,0.5,3.5,68,0.0,0.0,1025.25,122.4,20.7,68,3.5,16.8,16.8,11.0,217,0.10,544.96,18.2,331,0.4,67,0.0,0.0
01/09/18,09:10,16.9,89,15.1,0.9,3.5,68,0.0,0.0,1025.28,122.4,20.8,69,2.6,16.9,16.9,10.9,206,0.10,544.96,18.3,345,0.4,67,0.0,0.0
01/09/18,09:15,17.1,89,15.2,0.9,5.2,68,0.0,0.0,1025.35,122.4,20.8,69,5.2,17.1,17.1,10.9,152,0.10,544.96,18.4,359,0.4,67,0.0,0.0
01/09/18,09:30,17.7,88,15.7,0.9,3.5,73,0.0,0.0,1025.52,122.4,20.9,70,0.9,17.7,17.7,10.8,167,0.10,544.96,19.2,401,0.4,48,0.0,0.0
01/09/18,09:45,18.4,87,16.2,1.3,5.2,97,0.0,0.0,1025.58,122.4,21.2,70,0.9,18.4,18.4,10.7,250,0.10,544.96,20.0,441,0.4,92,0.0,0.0
01/09/18,10:00,19.3,85,16.7,1.3,3.5,178,0.0,0.0,1025.58,122.4,21.8,69,1.7,19.3,19.3,10.6,283,0.10,544.96,21.1,479,0.4,184,0.0,0.0
01/09/18,10:15,19.6,83,16.6,1.4,3.5,158,0.0,0.0,1025.62,122.4,22.3,68,0.9,19.6,19.6,10.7,309,0.23,545.08,21.3,516,0.4,240,0.0,0.0
01/09/18,10:30,19.9,80,16.3,1.0,4.3,167,0.0,0.0,1025.62,122.4,22.6,66,0.9,19.9,19.9,10.7,327,0.23,545.08,21.7,549,0.4,108,0.0,0.0
01/09/18,10:45,20.1,79,16.3,1.2,6.1,163,0.0,0.0,1025.72,122.4,22.9,65,0.9,20.1,20.1,10.9,341,0.23,545.08,21.8,581,0.4,117,0.0,0.0
Which converts to ..
"2018-09-01 09:00:00","16.7","90","15.1","0.3","3.8","99","0.0","0.0","1025.18","122.4","20.7","67","0.0","16.7","16.7","11.0","218","2.03","544.86","18.3","317","0.4","0","0.0","0.0","N","N"
"2018-09-01 09:05:00","16.8","90","15.1","0.7","5.1","68","0.0","0.0","1025.25","122.4","20.7","68","5.1","16.8","16.8","11.0","217","0.10","544.96","18.2","331","0.4","67","0.0","0.0","N","NE"
-
Converter will now handle Cumulus month log data from ver 1.8.4 upto mx
Missing data is replaced by 0.0 or 0 or "N" as appropriate.
@woolfg ......
What's happening ? - It was your plight that made the beteljuice do this in the first place ?
-
Converter will now handle Cumulus month log data from ver 1.8.4 upto mx
Missing data is replaced by 0.0 or 0 or "N" as appropriate.
@woolfg ......
What's happening ? - It was your plight that made the beteljuice do this in the first place ?
Hi
I am slowly getting my data imported . I have had a load of help from Lightmaster who looks after the Weather34 templates so between the both of you I am getting there. I just wish it wouldn't take so long to import each file
Thanks for your work too
Graham
-
New, improved, blah, blah, blah.
Now: Multiple file selection allowed - ie. marking a year of logs will produce ONE large 'import' file.
Cu > MB import converter (http://www.beteljuice.co.uk/DEMOS/convert.php)
-
This has kinda become more of a "Meteobridge Pro/NanoSD import issues" thread than a specifically cumulus one, but whatever, lol.
Found out from helping woolfg and txweather, that you must not have any duplicate sensors per timestamp, and timestamps must be in chronological order. With a Cumulus monthly file, you won't have duplicate sensors per timestamp unless you have 2 identical timestamps, however with a direct file which is a bunch of rows of "epoch-timestamp sensor data", you can accidentally create duplicate sensors.
If your getting an error about timestamps, and you know the timestamp is in the correct format for your import type, look above and below the row the system log complains about to see if it's all in the right order.
Also, if manually fixing files, DO NOT OPEN IN EXCEL unless you really know what you're doing. Excel likes to screw with dates and times and will change your timestamps. I've become very fond of VS Code, it has no issue opening .CSV and .TXT files with tens of millions of rows. (Excel maxes out a little over 1million rows).
-
If your getting an error about timestamps, and you know the timestamp is in the correct format for your import type, look above and below the row the system log complains about to see if it's all in the right order.
So ... are you saying if multiple Cumulus logs have been stitched together into one file and eg. Jan follows May everything will fold ?
-
the beteljuice, Can you give an option to import a raw file in imperial and export as a direct file in metric?
Even though Boris said it works it actually does not, lightmaster found that the files have to be in metric regardless.
And light has been awesome to help me convert but the problem has turned out that sometimes the timestamp is out of order and without some sanity checking the files will not fully import so it becomes painfully to fix manually.
I posted above how my data looks in meteohub raw files.
Thanks for your help!
-
Seems that there is a limit on how many records can be imported at a single time. Woolfg's yearly files stop importing on Oct 30th, but importing those last few months by themselves works.
Tx, if you're referring to your raw files, those are in metric, just have a decimal place without the actually period, ie: 0248 is 24.8C some other values however are integer values like the wind direction. The thorough documentation explains what it all is.
To summarize, so far, the rules are chronological order only, no duplicate sensors per timestamp, an unknown limit to how many records will import at once, and metric only (one of the formats has a value for if it's imperial, so that one may not have to be metric).
Also, I personally prefer the direct format because you can specify exactly which sensors the data goes to if you have multiple sensors (easily import data to the Pro's internal sensor).
-
Seems that there is a limit on how many records can be imported at a single time. Woolfg's yearly files stop importing on Oct 30th, but importing those last few months by themselves works.
Tx, if you're referring to your raw files, those are in metric, just have a decimal place without the actually period, ie: 0248 is 24.8C some other values however are integer values like the wind direction. The thorough documentation explains what it all is.
To summarize, so far, the rules are chronological order only, no duplicate sensors per timestamp, an unknown limit to how many records will import at once, and metric only (one of the formats has a value for if it's imperial, so that one may not have to be metric).
Also, I personally prefer the direct format because you can specify exactly which sensors the data goes to if you have multiple sensors (easily import data to the Pro's internal sensor).
Thanks for the correction light! I am tired asf. been awake for 24hrs :(
-
the beteljuice, Can you give an option to import a raw file in imperial and export as a direct file in metric?
Even though Boris said it works it actually does not, lightmaster found that the files have to be in metric regardless.
The formatted o/p file is in metric (regardless ..) - you just tell the prog what your native units are.
What do you mean by 'direct file' - The I/P file should be a Cumulus month log and the o/p file is in MB format, complete with 'falsified' zero values for differences in field length in Cumulus over the years. plus the two cardinal points at the end are calculated.
.. or are you talking about a non-cumulus format ???
@lightmaster .. So it's safer to just import one month at a time ?
Edit: beteljuice Cumulus converter modified - multiple files will be kept separate in one 'zip' file.
-
Arggh !
After re-reading the MB site tutorial more closely realised I incorrectly set wind units to km/h instead of m/s ](*,)
IMPORTANT - Anyone who has used the beteljuice converter should remove the data from MB and re-do again from the converter (now changed)
Sorry everyone - should have known it was going too well :oops: #-o
Cu > MB import converter (http://www.beteljuice.co.uk/DEMOS/convert.php)
-
Yes, monthly files seem to be best. You could probably get away with quarterly files, but they seem to take longer to import a few large files than it does to import many small files, probably due to limited system resources. I remember reading something that said all imported files are converted into direct files inside the system, then imported into the SQLite3 database. Presumably that happens in RAM which is limited, so a large yearly file would run out of RAM when it's converted into a direct file. The direct files are much larger than other files because the time stamp and sensor is written explicitly for each bit of data, so whereas a raw file would have 1 line with 1 timestamp, wind0, and 3 or 4 bits of data on a line, in a direct file that becomes 3 or 4 lines each with timestamp sensor/datatype, and then data. Meteohub Raw File to MB direct file results in roughly doubling the filesize. A cumulus monthly file with all the sensors on a single line, that's easily quadrupled in size.
So yes, smaller monthly files are much safer.