Author Topic: Cumulus Data Import  (Read 3706 times)

0 Members and 1 Guest are viewing this topic.

Offline woolfg

  • Member
  • *
  • Posts: 44
Cumulus Data Import
« on: May 07, 2019, 10:06:03 AM »
Hi

I would like to import my Cumulus historical  data going back10 years or so into my Meteobridge SQL database

I have a Cumulus dayfile.txt but  I dont know how to write the SQL to get the data into the database

To see the format of the dayfile.txt I have attached a file that shows what the columns are and also a sample of the file.

I hope this would update my Almanac Data on my website

Any help would be much appreciated

Kind Regards


Offline txweather.org

  • Forecaster
  • *****
  • Posts: 1597
    • Texas Weather
Re: Cumulus Data Import
« Reply #1 on: May 07, 2019, 08:33:43 PM »
If you have meteobridge pro all you have to do is dump your cumulus data in to the import folder.... The kicker is going to be.... What does meteobridge pro expects the filename to be in the import folder.

For Meteohub I figure out that the filename had to be the date the file was generaed... IE: If I was importing raw data from 07/07/214 than the filename had to be 201407 just liek the raw folder in meteohub.

Edit: Heads up... I am currently importing 6yrs and one year alone took me a full day.
« Last Edit: May 07, 2019, 09:03:26 PM by txweather.org »

----
Davis Vantage Pro2 Plus +FARS|Meteobridge Nano SD|Meteohub|Meteobridge MR-3020|WU KTXSPRIN75/PWS JRARGWX75/CWOP EW2972/WBB TXWDVUE75/Blitzortung ID: 1142|AWEKAS: 12095
Donations are welcome: https://paypal.me/ffuentesb

Offline woolfg

  • Member
  • *
  • Posts: 44
Re: Cumulus Data Import
« Reply #2 on: May 08, 2019, 03:25:47 AM »
Hi

Thanks for the reply but I am confused

Cumulus only produces data files for each month - I dont seem to have any daily files in my data folder at all

Regards

Graham

Offline txweather.org

  • Forecaster
  • *****
  • Posts: 1597
    • Texas Weather
Re: Cumulus Data Import
« Reply #3 on: May 08, 2019, 02:34:06 PM »
Hi

Thanks for the reply but I am confused

Cumulus only produces data files for each month - I dont seem to have any daily files in my data folder at all

Regards

Graham

Ill post the instructions for you.

What version of Meteobridge Hardware you have?

----
Davis Vantage Pro2 Plus +FARS|Meteobridge Nano SD|Meteohub|Meteobridge MR-3020|WU KTXSPRIN75/PWS JRARGWX75/CWOP EW2972/WBB TXWDVUE75/Blitzortung ID: 1142|AWEKAS: 12095
Donations are welcome: https://paypal.me/ffuentesb

Offline woolfg

  • Member
  • *
  • Posts: 44
Re: Cumulus Data Import
« Reply #4 on: May 08, 2019, 04:40:40 PM »
Hi

Thanks for the reply but I am confused

Cumulus only produces data files for each month - I dont seem to have any daily files in my data folder at all

Regards

Graham

Ill post the instructions for you.

What version of Meteobridge Hardware you have?

Hi

Thats kind of you - thanks. I have a Meteobridge Pro Red

Kind Regards

Graham

Offline txweather.org

  • Forecaster
  • *****
  • Posts: 1597
    • Texas Weather
Re: Cumulus Data Import
« Reply #5 on: May 09, 2019, 10:00:05 PM »
grahm open your data share folder of your meteobridge pro and go to the folder "import" drop your cumulas data in there and go to the history tab of your meteobridge. Go to import and see if your see your data.

----
Davis Vantage Pro2 Plus +FARS|Meteobridge Nano SD|Meteohub|Meteobridge MR-3020|WU KTXSPRIN75/PWS JRARGWX75/CWOP EW2972/WBB TXWDVUE75/Blitzortung ID: 1142|AWEKAS: 12095
Donations are welcome: https://paypal.me/ffuentesb

Offline lightmaster

  • MB-HWS Template Dev
  • Forecaster
  • *****
  • Posts: 324
    • Baxley,GA Weather
Re: Cumulus Data Import
« Reply #6 on: May 10, 2019, 01:15:14 AM »
Graham, have you gotten any data from Cumulus to import into your Pro? I've been trying a handmade Cumulus month log with my Nano SD to no avail. All my data is in a custom database, so trying to convert it to something Meteobridge will understand.

Offline woolfg

  • Member
  • *
  • Posts: 44
Re: Cumulus Data Import
« Reply #7 on: May 10, 2019, 03:20:56 AM »
grahm open your data share folder of your meteobridge pro and go to the folder "import" drop your cumulas data in there and go to the history tab of your meteobridge. Go to import and see if your see your data.

Hi

Yes I do see my data - I have tried all different types of the files in my Cumulus data folder, the dayfile.txt, the monthly data file and even the NOAA monthly report files. I can see all the files , I click on the import button and then get errors as shown in the attached screenshot.

Kind Regards

Graham

Offline woolfg

  • Member
  • *
  • Posts: 44
Re: Cumulus Data Import
« Reply #8 on: May 10, 2019, 03:23:00 AM »
Graham, have you gotten any data from Cumulus to import into your Pro? I've been trying a handmade Cumulus month log with my Nano SD to no avail. All my data is in a custom database, so trying to convert it to something Meteobridge will understand.

Hi Lightmaster

No I havent - all my data is in flat files - I have just posted a reply to txweather which has more info.

Kind Regards

Graham

Offline lightmaster

  • MB-HWS Template Dev
  • Forecaster
  • *****
  • Posts: 324
    • Baxley,GA Weather
Re: Cumulus Data Import
« Reply #9 on: May 10, 2019, 03:27:29 AM »
Graham, have you gotten any data from Cumulus to import into your Pro? I've been trying a handmade Cumulus month log with my Nano SD to no avail. All my data is in a custom database, so trying to convert it to something Meteobridge will understand.

Hi Lightmaster

No I havent - all my data is in flat files - I have just posted a reply to txweather which has more info.

Kind Regards

Graham

Same error I get. I've tried changing the time stamp on the first line to every kind I can think of and it still gives an error about the first line's timestamp.

Offline Bushman

  • Forecaster
  • *****
  • Posts: 7549
    • Eagle Bay Weather
Re: Cumulus Data Import
« Reply #10 on: May 10, 2019, 03:32:26 AM »
Timestamp? Is this a Euro vs. NA format issue?  I've seen lots of systems fail due to incorrect time formats.
Need low cost IP monitoring?  http://wirelesstag.net/wta.aspx?link=NisJxz6FhUa4V67/cwCRWA or PM me for 50% off Wirelesstags!!

Offline lightmaster

  • MB-HWS Template Dev
  • Forecaster
  • *****
  • Posts: 324
    • Baxley,GA Weather
Re: Cumulus Data Import
« Reply #11 on: May 10, 2019, 03:34:06 AM »
Timestamp? Is this a Euro vs. NA format issue?  I've seen lots of systems fail due to incorrect time formats.

I've tried every combination of year month day hour minute I can think of and also tried adding seconds. Even tried the exact same style of timestamp that the NANO uses in the system log

Offline woolfg

  • Member
  • *
  • Posts: 44
Re: Cumulus Data Import
« Reply #12 on: May 10, 2019, 04:20:28 AM »
Timestamp? Is this a Euro vs. NA format issue?  I've seen lots of systems fail due to incorrect time formats.

I've tried every combination of year month day hour minute I can think of and also tried adding seconds. Even tried the exact same style of timestamp that the NANO uses in the system log

The only other solution I can think of is for some clever person to write a SQL query that will read a Cumulus data file and write it directly to the database that is set up in the Meteobridge settings. I think the monthly data file would be best as that has all the high/low data for temperature rainfall sunshine etc

Regards

Graham

Offline docbee

  • Moderator
  • Forecaster
  • *****
  • Posts: 853
    • smartbedded
Re: Cumulus Data Import
« Reply #13 on: May 11, 2019, 04:16:19 AM »
Meteobridge does not take daily data, it expects data with a timestamp like "2018-03-23 23:12:03"

Cumulus data is expected to look like this:
Code: [Select]
INSERT INTO `Monthly` (`LogDateTime`, `Temp`, `Humidity`, `Dewpoint`, `Windspeed`, `Windgust`, `Windbearing`, `RainRate`, `TodayRainSoFar`, `Pressure`, `Raincounter`, `InsideTemp`, `InsideHumidity`, `LatestWindGust`, `WindChill`, `HeatIndex`, `UVindex`, `SolarRad`, `Evapotrans`, `AnnualEvapTran`, `ApparentTemp`, `MaxSolarRad`, `HrsSunShine`, `CurrWindBearing`, `RG11rain`, `RainSinceMidnight`, `WindbearingSym`, `CurrWindBearingSym`) VALUES
('2018-11-30 22:45:00', '4.3', '86', '2.1', '4.0', '8.0', '223', '0.0', '1.2', '1001.66', '553.0', '21.2', '42', '2.0', '2.7', '4.3', '0.0', '0.0', '0.6', '560.4', '1.4', '0.0', '2.1', '213', '0.0', '1.2', 'SW', 'SSW'),
('2018-11-30 22:40:00', '4.3', '86', '2.1', '5.0', '12.0', '227', '0.0', '1.2', '1001.73', '553.0', '21.3', '42', '5.0', '2.3', '4.3', '0.0', '0.0', '0.6', '560.4', '1.1', '0.0', '2.1', '230', '0.0', '1.2', 'SW', 'SW'),
('2018-11-30 22:35:00', '4.3', '86', '2.1', '5.0', '12.0', '224', '0.0', '1.2', '1001.66', '553.0', '21.3', '42', '4.0', '2.3', '4.3', '0.0', '0.0', '0.6', '560.4', '1.1', '0.0', '2.1', '217', '0.0', '1.2', 'SW', 'SW'),
('2018-11-30 22:30:00', '4.3', '86', '2.1', '5.0', '12.0', '221', '0.0', '1.2', '1001.52', '553.0', '21.4', '42', '9.0', '2.3', '4.3', '0.0', '0.0', '0.6', '560.4', '1.1', '0.0', '2.1', '244', '0.0', '1.2', 'SW', 'WSW'),
('2018-11-30 22:25:00', '4.2', '86', '2.1', '5.0', '12.0', '218', '0.0', '1.2', '1001.56', '553.0', '21.5', '42', '3.0', '2.2', '4.2', '0.0', '0.0', '0.6', '560.4', '1.0', '0.0', '2.1', '250', '0.0', '1.2', 'SW', 'WSW'),
('2018-11-30 22:20:00', '4.1', '87', '2.1', '5.0', '12.0', '213', '0.0', '1.2', '1001.59', '553.0', '21.6', '42', '6.0', '2.1', '4.1', '0.0', '0.0', '0.6', '560.4', '0.9', '0.0', '2.1', '168', '0.0', '1.2', 'SSW', 'SSE'),
('2018-11-30 22:15:00', '4.1', '87', '2.1', '5.0', '12.0', '217', '0.0', '1.2', '1001.49', '553.0', '21.7', '41', '6.0', '2.1', '4.1', '0.0', '0.0', '0.6', '560.4', '0.9', '0.0', '2.1', '179', '0.0', '1.2', 'SW', 'S'),
('2018-11-30 22:10:00', '4.1', '86', '2.0', '4.0', '12.0', '218', '0.0', '1.2', '1001.52', '553.0', '21.7', '41', '2.0', '2.6', '4.1', '0.0', '0.0', '0.6', '560.4', '1.2', '0.0', '2.1', '248', '0.0', '1.2', 'SW', 'WSW'),

or this

Code: [Select]

"2016-04-20 23:15:00","6.6","89","4.9","0.0","1.0","254","0.0","0.0","1027.40","397.4","25.2","41","1.0","6.6","6.6","0.0","0.0","0.0","0.0","5.5","0.0","0.0","257","0.0","0.0","WSW","WSW"
"2016-04-20 23:20:00","6.4","89","4.8","0.0","1.0","256","0.0","0.0","1027.43","397.4","25.2","41","1.0","6.4","6.4","0.0","0.0","0.0","0.0","5.3","0.0","0.0","255","0.0","0.0","WSW","WSW"
"2016-04-20 23:25:00","6.3","89","4.7","0.0","1.0","256","0.0","0.0","1027.53","397.4","25.2","41","0.0","6.3","6.3","0.0","0.0","0.0","0.0","5.1","0.0","0.0","256","0.0","0.0","WSW","WSW"
"2016-04-20 23:30:00","6.3","89","4.6","0.0","2.0","259","0.0","0.0","1027.53","397.4","25.2","41","0.0","6.3","6.3","0.0","0.0","0.0","0.0","5.1","0.0","0.0","264","0.0","0.0","W","W"
"2016-04-20 23:35:00","6.2","89","4.5","1.0","2.0","264","0.0","0.0","1027.70","397.4","25.2","41","0.0","6.2","6.2","0.0","0.0","0.0","0.0","4.7","0.0","0.0","273","0.0","0.0","W","W"
"2016-04-20 23:40:00","6.2","89","4.5","1.0","2.0","263","0.0","0.0","1027.74","397.4","25.2","41","2.0","6.2","6.2","0.0","0.0","0.0","0.0","4.6","0.0","0.0","270","0.0","0.0","W","W"
"2016-04-20 23:45:00","6.2","89","4.5","0.0","2.0","260","0.0","0.0","1027.70","397.4","25.2","41","0.0","6.2","6.2","0.0","0.0","0.0","0.0","4.9","0.0","0.0","270","0.0","0.0","W","W"
"2016-04-20 23:50:00","6.2","89","4.5","0.0","2.0","270","0.0","0.0","1027.63","397.4","25.1","41","0.0","6.2","6.2","0.0","0.0","0.0","0.0","4.9","0.0","0.0","270","0.0","0.0","W","W"
"2016-04-20 23:55:00","6.1","89","4.4","0.0","1.0","270","0.0","0.0","1027.67","397.4","25.2","41","0.0","6.1","6.1","0.0","0.0","0.0","0.0","4.9","0.0","0.0","270","0.0","0.0","W","W"
"2016-04-21 00:00:00","6.1","89","4.4","0.0","1.0","270","0.0","0.0","1027.74","397.4","25.1","41","0.0","6.1","6.1","0.0","0.0","0.0","0.0","4.9","0.0","0.0","270","0.0","0.0","W","W"
"2016-04-21 00:05:00","6.1","89","4.4","0.0","1.0","270","0.0","0.0","1027.84","397.4","25.1","40","0.0","6.1","6.1","0.0","0.0","0.0","0.0","4.8","0.0","0.0","270","0.0","0.0","W","W"
"2016-04-21 00:10:00","6.1","89","4.4","0.0","1.0","270","0.0","0.0","1027.87","397.4","25.1","40","0.0","6.1","6.1","0.0","0.0","0.0","0.0","4.8","0.0","0.0","270","0.0","0.0","W","W"
"2016-04-21 00:15:00","5.9","88","4.1","0.0","1.0","270","0.0","0.0","1027.87","397.4","25.1","40","0.0","5.9","5.9","0.0","0.0","0.0","0.0","4.6","0.0","0.0","269","0.0","0.0","W","W"
"2016-04-21 00:20:00","5.8","88","4.0","0.0","0.0","0","0.0","0.0","1027.87","397.4","25.0","40","0.0","5.8","5.8","0.0","0.0","0.0","0.0","4.5","0.0","0.0","270","0.0","0.0","N","W"
« Last Edit: May 11, 2019, 04:18:28 AM by docbee »
founder of smartbedded.com - home of meteohub, meteoplug, meteobridge, meteostick

Offline woolfg

  • Member
  • *
  • Posts: 44
Re: Cumulus Data Import
« Reply #14 on: May 11, 2019, 04:46:26 AM »
Hi Docbee

Thanks for the reply.

It looks like a lot of work to convert the Cumulus data files into a format that the Weather34 MySQL database would understand which is a shame.

If I reinstated my Envoy Data logger and Cumulus software and then connected it to my MeteoBridge Pro would my historical data get uploaded that way

Regards

Graham

Offline lightmaster

  • MB-HWS Template Dev
  • Forecaster
  • *****
  • Posts: 324
    • Baxley,GA Weather
Re: Cumulus Data Import
« Reply #15 on: May 11, 2019, 04:55:01 AM »
Hi Docbee

Thanks for the reply.

It looks like a lot of work to convert the Cumulus data files into a format that the Weather34 MySQL database would understand which is a shame.

If I reinstated my Envoy Data logger and Cumulus software and then connected it to my MeteoBridge Pro would my historical data get uploaded that way

Regards

Graham

I'm assuming you meant the Meteobridge SQLite database... As far as I can tell, only difference seems be be the data time format. Gonna try to import a month of mine in a bit when I get a chance.



Meteobridge does not take daily data, it expects data with a timestamp like "2018-03-23 23:12:03"

Cumulus data is expected to look like this:
Code: [Select]
INSERT INTO `Monthly` (`LogDateTime`, `Temp`, `Humidity`, `Dewpoint`, `Windspeed`, `Windgust`, `Windbearing`, `RainRate`, `TodayRainSoFar`, `Pressure`, `Raincounter`, `InsideTemp`, `InsideHumidity`, `LatestWindGust`, `WindChill`, `HeatIndex`, `UVindex`, `SolarRad`, `Evapotrans`, `AnnualEvapTran`, `ApparentTemp`, `MaxSolarRad`, `HrsSunShine`, `CurrWindBearing`, `RG11rain`, `RainSinceMidnight`, `WindbearingSym`, `CurrWindBearingSym`) VALUES
('2018-11-30 22:45:00', '4.3', '86', '2.1', '4.0', '8.0', '223', '0.0', '1.2', '1001.66', '553.0', '21.2', '42', '2.0', '2.7', '4.3', '0.0', '0.0', '0.6', '560.4', '1.4', '0.0', '2.1', '213', '0.0', '1.2', 'SW', 'SSW'),
('2018-11-30 22:40:00', '4.3', '86', '2.1', '5.0', '12.0', '227', '0.0', '1.2', '1001.73', '553.0', '21.3', '42', '5.0', '2.3', '4.3', '0.0', '0.0', '0.6', '560.4', '1.1', '0.0', '2.1', '230', '0.0', '1.2', 'SW', 'SW'),
('2018-11-30 22:35:00', '4.3', '86', '2.1', '5.0', '12.0', '224', '0.0', '1.2', '1001.66', '553.0', '21.3', '42', '4.0', '2.3', '4.3', '0.0', '0.0', '0.6', '560.4', '1.1', '0.0', '2.1', '217', '0.0', '1.2', 'SW', 'SW'),
('2018-11-30 22:30:00', '4.3', '86', '2.1', '5.0', '12.0', '221', '0.0', '1.2', '1001.52', '553.0', '21.4', '42', '9.0', '2.3', '4.3', '0.0', '0.0', '0.6', '560.4', '1.1', '0.0', '2.1', '244', '0.0', '1.2', 'SW', 'WSW'),
('2018-11-30 22:25:00', '4.2', '86', '2.1', '5.0', '12.0', '218', '0.0', '1.2', '1001.56', '553.0', '21.5', '42', '3.0', '2.2', '4.2', '0.0', '0.0', '0.6', '560.4', '1.0', '0.0', '2.1', '250', '0.0', '1.2', 'SW', 'WSW'),
('2018-11-30 22:20:00', '4.1', '87', '2.1', '5.0', '12.0', '213', '0.0', '1.2', '1001.59', '553.0', '21.6', '42', '6.0', '2.1', '4.1', '0.0', '0.0', '0.6', '560.4', '0.9', '0.0', '2.1', '168', '0.0', '1.2', 'SSW', 'SSE'),
('2018-11-30 22:15:00', '4.1', '87', '2.1', '5.0', '12.0', '217', '0.0', '1.2', '1001.49', '553.0', '21.7', '41', '6.0', '2.1', '4.1', '0.0', '0.0', '0.6', '560.4', '0.9', '0.0', '2.1', '179', '0.0', '1.2', 'SW', 'S'),
('2018-11-30 22:10:00', '4.1', '86', '2.0', '4.0', '12.0', '218', '0.0', '1.2', '1001.52', '553.0', '21.7', '41', '2.0', '2.6', '4.1', '0.0', '0.0', '0.6', '560.4', '1.2', '0.0', '2.1', '248', '0.0', '1.2', 'SW', 'WSW'),

or this

Code: [Select]

"2016-04-20 23:15:00","6.6","89","4.9","0.0","1.0","254","0.0","0.0","1027.40","397.4","25.2","41","1.0","6.6","6.6","0.0","0.0","0.0","0.0","5.5","0.0","0.0","257","0.0","0.0","WSW","WSW"
"2016-04-20 23:20:00","6.4","89","4.8","0.0","1.0","256","0.0","0.0","1027.43","397.4","25.2","41","1.0","6.4","6.4","0.0","0.0","0.0","0.0","5.3","0.0","0.0","255","0.0","0.0","WSW","WSW"
"2016-04-20 23:25:00","6.3","89","4.7","0.0","1.0","256","0.0","0.0","1027.53","397.4","25.2","41","0.0","6.3","6.3","0.0","0.0","0.0","0.0","5.1","0.0","0.0","256","0.0","0.0","WSW","WSW"
"2016-04-20 23:30:00","6.3","89","4.6","0.0","2.0","259","0.0","0.0","1027.53","397.4","25.2","41","0.0","6.3","6.3","0.0","0.0","0.0","0.0","5.1","0.0","0.0","264","0.0","0.0","W","W"
"2016-04-20 23:35:00","6.2","89","4.5","1.0","2.0","264","0.0","0.0","1027.70","397.4","25.2","41","0.0","6.2","6.2","0.0","0.0","0.0","0.0","4.7","0.0","0.0","273","0.0","0.0","W","W"
"2016-04-20 23:40:00","6.2","89","4.5","1.0","2.0","263","0.0","0.0","1027.74","397.4","25.2","41","2.0","6.2","6.2","0.0","0.0","0.0","0.0","4.6","0.0","0.0","270","0.0","0.0","W","W"
"2016-04-20 23:45:00","6.2","89","4.5","0.0","2.0","260","0.0","0.0","1027.70","397.4","25.2","41","0.0","6.2","6.2","0.0","0.0","0.0","0.0","4.9","0.0","0.0","270","0.0","0.0","W","W"
"2016-04-20 23:50:00","6.2","89","4.5","0.0","2.0","270","0.0","0.0","1027.63","397.4","25.1","41","0.0","6.2","6.2","0.0","0.0","0.0","0.0","4.9","0.0","0.0","270","0.0","0.0","W","W"
"2016-04-20 23:55:00","6.1","89","4.4","0.0","1.0","270","0.0","0.0","1027.67","397.4","25.2","41","0.0","6.1","6.1","0.0","0.0","0.0","0.0","4.9","0.0","0.0","270","0.0","0.0","W","W"
"2016-04-21 00:00:00","6.1","89","4.4","0.0","1.0","270","0.0","0.0","1027.74","397.4","25.1","41","0.0","6.1","6.1","0.0","0.0","0.0","0.0","4.9","0.0","0.0","270","0.0","0.0","W","W"
"2016-04-21 00:05:00","6.1","89","4.4","0.0","1.0","270","0.0","0.0","1027.84","397.4","25.1","40","0.0","6.1","6.1","0.0","0.0","0.0","0.0","4.8","0.0","0.0","270","0.0","0.0","W","W"
"2016-04-21 00:10:00","6.1","89","4.4","0.0","1.0","270","0.0","0.0","1027.87","397.4","25.1","40","0.0","6.1","6.1","0.0","0.0","0.0","0.0","4.8","0.0","0.0","270","0.0","0.0","W","W"
"2016-04-21 00:15:00","5.9","88","4.1","0.0","1.0","270","0.0","0.0","1027.87","397.4","25.1","40","0.0","5.9","5.9","0.0","0.0","0.0","0.0","4.6","0.0","0.0","269","0.0","0.0","W","W"
"2016-04-21 00:20:00","5.8","88","4.0","0.0","0.0","0","0.0","0.0","1027.87","397.4","25.0","40","0.0","5.8","5.8","0.0","0.0","0.0","0.0","4.5","0.0","0.0","270","0.0","0.0","N","W"

Does data have to be in metric, or is MB capable of recognizing the difference? Obviously if temp is 90, its not Celcius, lol. If it has to be in metric, not the end of the world for me, just gotta export it from my database again but with metric conversions in it.

Offline lightmaster

  • MB-HWS Template Dev
  • Forecaster
  • *****
  • Posts: 324
    • Baxley,GA Weather
Re: Cumulus Data Import
« Reply #16 on: May 11, 2019, 05:03:59 AM »
Combining the date and time columns as YYYY-MM-DD hh:mm:ss worked. File doesn't have seconds, so I just added 00 as the seconds.

Offline woolfg

  • Member
  • *
  • Posts: 44
Re: Cumulus Data Import
« Reply #17 on: May 11, 2019, 05:11:50 AM »
Combining the date and time columns as YYYY-MM-DD hh:mm:ss worked. File does have seconds, so I just added 00 as the seconds.

Hi Lightmaster

OK thats encouraging - which cumulus file did you use . I have filenames like FEB19log.txt. Also how did you convert the date/time - is there a quick way of doing it.

Regards

Graham

Offline docbee

  • Moderator
  • Forecaster
  • *****
  • Posts: 853
    • smartbedded
Re: Cumulus Data Import
« Reply #18 on: May 11, 2019, 05:24:53 AM »
I just added a section about data imports to the wiki. That might help as well.
https://www.meteobridge.com/wiki/index.php/PRO_Tutorial#Data_Import
founder of smartbedded.com - home of meteohub, meteoplug, meteobridge, meteostick

Offline lightmaster

  • MB-HWS Template Dev
  • Forecaster
  • *****
  • Posts: 324
    • Baxley,GA Weather
Re: Cumulus Data Import
« Reply #19 on: May 11, 2019, 05:35:45 AM »
I just added a section about data imports to the wiki. That might help as well.
https://www.meteobridge.com/wiki/index.php/PRO_Tutorial#Data_Import

Documentation on the import capabilities looks great. Might wanna add a link in the Nano SD and Pro product pages to direct people to that documentation.

Offline woolfg

  • Member
  • *
  • Posts: 44
Re: Cumulus Data Import
« Reply #20 on: May 11, 2019, 05:37:48 AM »
I just added a section about data imports to the wiki. That might help as well.
https://www.meteobridge.com/wiki/index.php/PRO_Tutorial#Data_Import

Hi Docbee

Thanks for that - unfortunately my cumulus data files have hundreds of lines in them and I can see no easy way of converting them unless I do it manually. Mine look like this

01/09/18,09:00,16.7,90,15.1,0.2,2.6,99,0.0,0.0,1025.18,122.4,20.7,67,0.0,16.7,16.7,11.0,218,2.03,544.86,18.3,317,0.4,0,0.0,0.0
01/09/18,09:05,16.8,90,15.1,0.5,3.5,68,0.0,0.0,1025.25,122.4,20.7,68,3.5,16.8,16.8,11.0,217,0.10,544.96,18.2,331,0.4,67,0.0,0.0
01/09/18,09:10,16.9,89,15.1,0.9,3.5,68,0.0,0.0,1025.28,122.4,20.8,69,2.6,16.9,16.9,10.9,206,0.10,544.96,18.3,345,0.4,67,0.0,0.0
01/09/18,09:15,17.1,89,15.2,0.9,5.2,68,0.0,0.0,1025.35,122.4,20.8,69,5.2,17.1,17.1,10.9,152,0.10,544.96,18.4,359,0.4,67,0.0,0.0
01/09/18,09:30,17.7,88,15.7,0.9,3.5,73,0.0,0.0,1025.52,122.4,20.9,70,0.9,17.7,17.7,10.8,167,0.10,544.96,19.2,401,0.4,48,0.0,0.0
01/09/18,09:45,18.4,87,16.2,1.3,5.2,97,0.0,0.0,1025.58,122.4,21.2,70,0.9,18.4,18.4,10.7,250,0.10,544.96,20.0,441,0.4,92,0.0,0.0
01/09/18,10:00,19.3,85,16.7,1.3,3.5,178,0.0,0.0,1025.58,122.4,21.8,69,1.7,19.3,19.3,10.6,283,0.10,544.96,21.1,479,0.4,184,0.0,0.0
01/09/18,10:15,19.6,83,16.6,1.4,3.5,158,0.0,0.0,1025.62,122.4,22.3,68,0.9,19.6,19.6,10.7,309,0.23,545.08,21.3,516,0.4,240,0.0,0.0
01/09/18,10:30,19.9,80,16.3,1.0,4.3,167,0.0,0.0,1025.62,122.4,22.6,66,0.9,19.9,19.9,10.7,327,0.23,545.08,21.7,549,0.4,108,0.0,0.0
01/09/18,10:45,20.1,79,16.3,1.2,6.1,163,0.0,0.0,1025.72,122.4,22.9,65,0.9,20.1,20.1,10.9,341,0.23,545.08,21.8,581,0.4,117,0.0,0.0

I dont have the expertise either in Excel of programming to write a utility that would ease the conversion
Regards

Graham

Offline lightmaster

  • MB-HWS Template Dev
  • Forecaster
  • *****
  • Posts: 324
    • Baxley,GA Weather
Re: Cumulus Data Import
« Reply #21 on: May 11, 2019, 05:43:06 AM »
I just added a section about data imports to the wiki. That might help as well.
https://www.meteobridge.com/wiki/index.php/PRO_Tutorial#Data_Import

Hi Docbee

Thanks for that - unfortunately my cumulus data files have hundreds of lines in them and I can see no easy way of converting them unless I do it manually. Mine look like this

01/09/18,09:00,16.7,90,15.1,0.2,2.6,99,0.0,0.0,1025.18,122.4,20.7,67,0.0,16.7,16.7,11.0,218,2.03,544.86,18.3,317,0.4,0,0.0,0.0
01/09/18,09:05,16.8,90,15.1,0.5,3.5,68,0.0,0.0,1025.25,122.4,20.7,68,3.5,16.8,16.8,11.0,217,0.10,544.96,18.2,331,0.4,67,0.0,0.0
01/09/18,09:10,16.9,89,15.1,0.9,3.5,68,0.0,0.0,1025.28,122.4,20.8,69,2.6,16.9,16.9,10.9,206,0.10,544.96,18.3,345,0.4,67,0.0,0.0
01/09/18,09:15,17.1,89,15.2,0.9,5.2,68,0.0,0.0,1025.35,122.4,20.8,69,5.2,17.1,17.1,10.9,152,0.10,544.96,18.4,359,0.4,67,0.0,0.0
01/09/18,09:30,17.7,88,15.7,0.9,3.5,73,0.0,0.0,1025.52,122.4,20.9,70,0.9,17.7,17.7,10.8,167,0.10,544.96,19.2,401,0.4,48,0.0,0.0
01/09/18,09:45,18.4,87,16.2,1.3,5.2,97,0.0,0.0,1025.58,122.4,21.2,70,0.9,18.4,18.4,10.7,250,0.10,544.96,20.0,441,0.4,92,0.0,0.0
01/09/18,10:00,19.3,85,16.7,1.3,3.5,178,0.0,0.0,1025.58,122.4,21.8,69,1.7,19.3,19.3,10.6,283,0.10,544.96,21.1,479,0.4,184,0.0,0.0
01/09/18,10:15,19.6,83,16.6,1.4,3.5,158,0.0,0.0,1025.62,122.4,22.3,68,0.9,19.6,19.6,10.7,309,0.23,545.08,21.3,516,0.4,240,0.0,0.0
01/09/18,10:30,19.9,80,16.3,1.0,4.3,167,0.0,0.0,1025.62,122.4,22.6,66,0.9,19.9,19.9,10.7,327,0.23,545.08,21.7,549,0.4,108,0.0,0.0
01/09/18,10:45,20.1,79,16.3,1.2,6.1,163,0.0,0.0,1025.72,122.4,22.9,65,0.9,20.1,20.1,10.9,341,0.23,545.08,21.8,581,0.4,117,0.0,0.0

I dont have the expertise either in Excel of programming to write a utility that would ease the conversion
Regards

Graham

Yours are all in metric already, at least that I can see. Download VSCode (a coding text editor). If you open a file and then press Ctrl+Alt+down arrow, keep pressing the down arrow and you can have a cursor on every line, then using cut and paste, you can rearrange an entire file at once so it's in the right order. Typing and delete also work like this, so you can delete the extra comma and type in the 20 in 2019, 2018, etc.

Basically, edit one line and the entire file is edited

Offline Bushman

  • Forecaster
  • *****
  • Posts: 7549
    • Eagle Bay Weather
Re: Cumulus Data Import
« Reply #22 on: May 11, 2019, 08:46:35 PM »
I just added a section about data imports to the wiki. That might help as well.
https://www.meteobridge.com/wiki/index.php/PRO_Tutorial#Data_Import

Hi Docbee

Thanks for that - unfortunately my cumulus data files have hundreds of lines in them and I can see no easy way of converting them unless I do it manually. Mine look like this

01/09/18,09:00,16.7,90,15.1,0.2,2.6,99,0.0,0.0,1025.18,122.4,20.7,67,0.0,16.7,16.7,11.0,218,2.03,544.86,18.3,317,0.4,0,0.0,0.0
01/09/18,09:05,16.8,90,15.1,0.5,3.5,68,0.0,0.0,1025.25,122.4,20.7,68,3.5,16.8,16.8,11.0,217,0.10,544.96,18.2,331,0.4,67,0.0,0.0
01/09/18,09:10,16.9,89,15.1,0.9,3.5,68,0.0,0.0,1025.28,122.4,20.8,69,2.6,16.9,16.9,10.9,206,0.10,544.96,18.3,345,0.4,67,0.0,0.0
01/09/18,09:15,17.1,89,15.2,0.9,5.2,68,0.0,0.0,1025.35,122.4,20.8,69,5.2,17.1,17.1,10.9,152,0.10,544.96,18.4,359,0.4,67,0.0,0.0
01/09/18,09:30,17.7,88,15.7,0.9,3.5,73,0.0,0.0,1025.52,122.4,20.9,70,0.9,17.7,17.7,10.8,167,0.10,544.96,19.2,401,0.4,48,0.0,0.0
01/09/18,09:45,18.4,87,16.2,1.3,5.2,97,0.0,0.0,1025.58,122.4,21.2,70,0.9,18.4,18.4,10.7,250,0.10,544.96,20.0,441,0.4,92,0.0,0.0
01/09/18,10:00,19.3,85,16.7,1.3,3.5,178,0.0,0.0,1025.58,122.4,21.8,69,1.7,19.3,19.3,10.6,283,0.10,544.96,21.1,479,0.4,184,0.0,0.0
01/09/18,10:15,19.6,83,16.6,1.4,3.5,158,0.0,0.0,1025.62,122.4,22.3,68,0.9,19.6,19.6,10.7,309,0.23,545.08,21.3,516,0.4,240,0.0,0.0
01/09/18,10:30,19.9,80,16.3,1.0,4.3,167,0.0,0.0,1025.62,122.4,22.6,66,0.9,19.9,19.9,10.7,327,0.23,545.08,21.7,549,0.4,108,0.0,0.0
01/09/18,10:45,20.1,79,16.3,1.2,6.1,163,0.0,0.0,1025.72,122.4,22.9,65,0.9,20.1,20.1,10.9,341,0.23,545.08,21.8,581,0.4,117,0.0,0.0

I dont have the expertise either in Excel of programming to write a utility that would ease the conversion
Regards

Graham

Is this a one-time thing for you?  Or are you planning on regularly importing these monthly files?  It is pretty simple to do in Excel (although I'd probably use AWK (GAWK for Windows).  Another alternative is Notepad++   BUt if you are doing this regularly, a simple piece of code could be written.
Need low cost IP monitoring?  http://wirelesstag.net/wta.aspx?link=NisJxz6FhUa4V67/cwCRWA or PM me for 50% off Wirelesstags!!

Offline txweather.org

  • Forecaster
  • *****
  • Posts: 1597
    • Texas Weather
Re: Cumulus Data Import
« Reply #23 on: May 11, 2019, 09:22:31 PM »
I just added a section about data imports to the wiki. That might help as well.
https://www.meteobridge.com/wiki/index.php/PRO_Tutorial#Data_Import

Hi Docbee

Thanks for that - unfortunately my cumulus data files have hundreds of lines in them and I can see no easy way of converting them unless I do it manually. Mine look like this

01/09/18,09:00,16.7,90,15.1,0.2,2.6,99,0.0,0.0,1025.18,122.4,20.7,67,0.0,16.7,16.7,11.0,218,2.03,544.86,18.3,317,0.4,0,0.0,0.0
01/09/18,09:05,16.8,90,15.1,0.5,3.5,68,0.0,0.0,1025.25,122.4,20.7,68,3.5,16.8,16.8,11.0,217,0.10,544.96,18.2,331,0.4,67,0.0,0.0
01/09/18,09:10,16.9,89,15.1,0.9,3.5,68,0.0,0.0,1025.28,122.4,20.8,69,2.6,16.9,16.9,10.9,206,0.10,544.96,18.3,345,0.4,67,0.0,0.0
01/09/18,09:15,17.1,89,15.2,0.9,5.2,68,0.0,0.0,1025.35,122.4,20.8,69,5.2,17.1,17.1,10.9,152,0.10,544.96,18.4,359,0.4,67,0.0,0.0
01/09/18,09:30,17.7,88,15.7,0.9,3.5,73,0.0,0.0,1025.52,122.4,20.9,70,0.9,17.7,17.7,10.8,167,0.10,544.96,19.2,401,0.4,48,0.0,0.0
01/09/18,09:45,18.4,87,16.2,1.3,5.2,97,0.0,0.0,1025.58,122.4,21.2,70,0.9,18.4,18.4,10.7,250,0.10,544.96,20.0,441,0.4,92,0.0,0.0
01/09/18,10:00,19.3,85,16.7,1.3,3.5,178,0.0,0.0,1025.58,122.4,21.8,69,1.7,19.3,19.3,10.6,283,0.10,544.96,21.1,479,0.4,184,0.0,0.0
01/09/18,10:15,19.6,83,16.6,1.4,3.5,158,0.0,0.0,1025.62,122.4,22.3,68,0.9,19.6,19.6,10.7,309,0.23,545.08,21.3,516,0.4,240,0.0,0.0
01/09/18,10:30,19.9,80,16.3,1.0,4.3,167,0.0,0.0,1025.62,122.4,22.6,66,0.9,19.9,19.9,10.7,327,0.23,545.08,21.7,549,0.4,108,0.0,0.0
01/09/18,10:45,20.1,79,16.3,1.2,6.1,163,0.0,0.0,1025.72,122.4,22.9,65,0.9,20.1,20.1,10.9,341,0.23,545.08,21.8,581,0.4,117,0.0,0.0

I dont have the expertise either in Excel of programming to write a utility that would ease the conversion
Regards

Graham

Is this a one-time thing for you?  Or are you planning on regularly importing these monthly files?  It is pretty simple to do in Excel (although I'd probably use AWK (GAWK for Windows).  Another alternative is Notepad++   BUt if you are doing this regularly, a simple piece of code could be written.

I am a bit lost with this whole import thing... Let me elaborate.... If I have a native Meteohub raw file.... Why do I have to convert the file to a format that the nano sd understands? This seems like something that could have been written on the import process instead of saying "you can import files from meteohub" when in reality you really cant...

I have a full year of meteohub data that I thought I was importing "fine" to the nano sd but it turns out that now my nano sd has a crap load of bad data fort hat whole year!

Here is a snippet of the meteohub raw data:

Code: [Select]
20180701000400 thb1 262 51 0153 10077 -1 10135
20180701000400 th0 273 87 0249
20180701000400 wind0 158 36 18 0273
20180701000400 rain0 0 0 1524
20180701000900 thb1 263 55 0165 10077 -1 10135
20180701000900 th0 273 87 0249
20180701000900 wind0 158 31 18 0273
20180701000900 rain0 0 0 1524
20180701001400 thb1 264 54 0163 10077 -1 10135
20180701001400 th0 273 87 0249
20180701001400 wind0 180 31 13 0273
20180701001400 rain0 0 0 1524
20180701001900 thb1 265 51 0155 10077 -1 10135
20180701001900 th0 273 87 0249
20180701001900 wind0 180 27 13 0273
20180701001900 rain0 0 0 1524
20180701002400 thb1 263 49 0147 10078 -1 10136
20180701002400 th0 272 87 0248
20180701002400 wind0 203 22 13 0272
20180701002400 rain0 0 0 1524
20180701002900 thb1 262 51 0153 10077 -1 10135
20180701002900 th0 272 87 0248
20180701002900 wind0 158 22 13 0272
20180701002900 rain0 0 0 1524
20180701003400 thb1 263 53 0160 10076 -1 10134
20180701003400 th0 272 87 0248
20180701003400 wind0 158 22 13 0272
20180701003400 rain0 0 0 1524
20180701003900 thb1 264 57 0172 10076 -1 10134
20180701003900 th0 271 87 0247
20180701003900 wind0 158 22 13 0271
20180701003900 rain0 0 0 1524

And for the year of 2018 everything that I imported is completely off temperature wise... Its saying July in Texas was 32F! LOL

To me it seem like is just not plain supported unless converted to what ever the nano expects.
« Last Edit: May 11, 2019, 09:24:11 PM by txweather.org »

----
Davis Vantage Pro2 Plus +FARS|Meteobridge Nano SD|Meteohub|Meteobridge MR-3020|WU KTXSPRIN75/PWS JRARGWX75/CWOP EW2972/WBB TXWDVUE75/Blitzortung ID: 1142|AWEKAS: 12095
Donations are welcome: https://paypal.me/ffuentesb

Offline the beteljuice

  • the beteljuice
  • Forecaster
  • *****
  • Posts: 316
    • test site
Re: Cumulus Data Import
« Reply #24 on: May 12, 2019, 07:49:23 PM »
the beteljuice needed a break from some truly astronomical math coding, so ......

Try Cumulus month log > MB import format

I have no data files nor weather software, so this should be regarded as risky !
It will convert date / time format, convert units where necessary and it will create a .txt file in the format of the above posts.

No idea if it will actually 'import' or not !!!

Edit: Now proven to 'import' (Thanks Kocher)

Edit: Handles Cu files from ver 1.8.4 > 1.9.5 and mx.
        Multiple file selection allowed - ie. marking a year of logs will produce ONE large 'import' file.
        (Any missing fields are 0, 0.0, or "N" as appropriate)

Use 'debug' on your first run so you can easily check if any conversions have been applied correctly.

Let me know if your PC catches fire .....  :shock:
« Last Edit: May 14, 2019, 05:46:48 PM by the beteljuice »
Imagine what you will KNOW tomorrow !