Author Topic: Cumulus Data Import  (Read 1189 times)

0 Members and 1 Guest are viewing this topic.

Offline PaulMy

  • Forecaster
  • *****
  • Posts: 4476
    • KomokaWeather
Re: Cumulus Data Import
« Reply #25 on: May 12, 2019, 08:01:51 PM »
It works... :thumbsup:

Enjoy,
Paul
Davis Vantage Pro 2 Plus 24-FARS Wireless
Envoy/USB DataLogger/Cumulus v.1.9.4   -   Vue Console/WiFiLogger/CumulusMX
Komoka, ON  Canada
www.komokaweather.ca   www.komokaweather.com   www.komokaweather.com/weather28/  
www.komokaweather.com/pws/index.php
Blitzortung Station #1076 www.lightningmaps.org/blitzortung7/america/index.php?bo_page=statistics&bo_show=network&lang=en&bo_station_id=969


Offline the beteljuice

  • the beteljuice
  • Contributor
  • ***
  • Posts: 128
    • test site
Re: Cumulus Data Import
« Reply #26 on: May 12, 2019, 08:33:06 PM »
.... but

... will it 'import' ?? - and you are already using the requested Units Of Measure.
« Last Edit: May 12, 2019, 08:35:08 PM by the beteljuice »
Imagine what you will KNOW tomorrow !

Offline PaulMy

  • Forecaster
  • *****
  • Posts: 4476
    • KomokaWeather
Re: Cumulus Data Import
« Reply #27 on: May 12, 2019, 09:17:40 PM »
...aaaaaaaaah
and the converted data is wrong #-o
Using the same metric units
Quote
"2019-05-01 00:00:00","5.7","77","1.9","12.9","36.4","1","0.0","0.0","1023.98","222.8","26.1","28","15.0","3.2","5.7","0.0","0","1.22","147.96","1.5","0","0.2","360","0.0","0.0","N","N"
Converting to Imperial units
Quote
"2019-05-01 00:00:00","-14.6","77","-16.7","18.9","53.4","1","0.0","0.0","34675.94","5659.1","-3.3","28","22.0","-16.0","-14.6","0.0","0","1.22","147.96","-16.9","0","0.2","360","0.0","0.0","N","N"

Enjoy,
Paul
Davis Vantage Pro 2 Plus 24-FARS Wireless
Envoy/USB DataLogger/Cumulus v.1.9.4   -   Vue Console/WiFiLogger/CumulusMX
Komoka, ON  Canada
www.komokaweather.ca   www.komokaweather.com   www.komokaweather.com/weather28/  
www.komokaweather.com/pws/index.php
Blitzortung Station #1076 www.lightningmaps.org/blitzortung7/america/index.php?bo_page=statistics&bo_show=network&lang=en&bo_station_id=969


Offline the beteljuice

  • the beteljuice
  • Contributor
  • ***
  • Posts: 128
    • test site
Re: Cumulus Data Import
« Reply #28 on: May 12, 2019, 09:34:49 PM »
Quote
and the converted data is wrong - Using the same metric units, ... Converting to Imperial units
???? It doesn't convert to Imperial ...

It asks you what your units are and converts them to the required ones.

This:
"2019-05-01 00:00:00","5.7","77","1.9", after being told it's Imperial
becomes
"2019-05-01 00:00:00","-14.6","77","-16.7", as metric

.. or am I misunderstanding you ?
Imagine what you will KNOW tomorrow !

Offline PaulMy

  • Forecaster
  • *****
  • Posts: 4476
    • KomokaWeather
Re: Cumulus Data Import
« Reply #29 on: May 12, 2019, 09:58:33 PM »
My Cumulus monthly source file is metric. 

In the first test I selected the metric units in your utility and that seemed to convert ok to metric to the MB format. =D>
My second test was wrong as I then selected the imperial units :oops:   

I don't have MB so don't know if the converted format file can be imported.

Enjoy,
Paul
Davis Vantage Pro 2 Plus 24-FARS Wireless
Envoy/USB DataLogger/Cumulus v.1.9.4   -   Vue Console/WiFiLogger/CumulusMX
Komoka, ON  Canada
www.komokaweather.ca   www.komokaweather.com   www.komokaweather.com/weather28/  
www.komokaweather.com/pws/index.php
Blitzortung Station #1076 www.lightningmaps.org/blitzortung7/america/index.php?bo_page=statistics&bo_show=network&lang=en&bo_station_id=969


Offline the beteljuice

  • the beteljuice
  • Contributor
  • ***
  • Posts: 128
    • test site
Re: Cumulus Data Import
« Reply #30 on: May 12, 2019, 10:44:47 PM »
Quote
I don't have MB so don't know if the converted format file can be imported.
Tsk Paul - I thought you had the betas of everything that's out there  :-P

... Anyone who can check if it 'imports' or not ????
Imagine what you will KNOW tomorrow !

Offline woolfg

  • Member
  • *
  • Posts: 26
Re: Cumulus Data Import
« Reply #31 on: May 13, 2019, 03:40:17 AM »
the beteljuice needed a break from some truly astronomical math coding, so ......

Try Cumulus month log > MB import format

I have no data files nor weather software, so this should be regarded as risky !
It should convert date / time format, convert units where necessary and it will create a .txt file in the format of the above posts.

No idea if it will actually 'import' or not !!!

Great - I'll give it a try

Use 'debug' on your first run so you can easily check if any conversions have been applied correctly.

Let me know if your PC catches fire .....  :shock:

Offline woolfg

  • Member
  • *
  • Posts: 26
Re: Cumulus Data Import
« Reply #32 on: May 13, 2019, 03:40:44 AM »
the beteljuice needed a break from some truly astronomical math coding, so ......

Try Cumulus month log > MB import format

I have no data files nor weather software, so this should be regarded as risky !
It should convert date / time format, convert units where necessary and it will create a .txt file in the format of the above posts.

No idea if it will actually 'import' or not !!!

Great - I'll give it a try

Use 'debug' on your first run so you can easily check if any conversions have been applied correctly.

Let me know if your PC catches fire .....  :shock:

I'll give it a try

Offline woolfg

  • Member
  • *
  • Posts: 26
Re: Cumulus Data Import
« Reply #33 on: May 13, 2019, 05:06:06 AM »
the beteljuice needed a break from some truly astronomical math coding, so ......

Try Cumulus month log > MB import format

I have no data files nor weather software, so this should be regarded as risky !
It should convert date / time format, convert units where necessary and it will create a .txt file in the format of the above posts.

No idea if it will actually 'import' or not !!!

Great - I'll give it a try

Use 'debug' on your first run so you can easily check if any conversions have been applied correctly.

Let me know if your PC catches fire .....  :shock:

I'll give it a try

It doesnt work for me -it seems to fail on the pressure for some reason. I select mb as my pressure data is 1017.34 for example but it doesnt seem to like that format

Offline kocher

  • Senior Member
  • **
  • Posts: 79
    • San Sebastian (Spain)
Re: Cumulus Data Import
« Reply #34 on: May 13, 2019, 06:30:47 AM »
Greetings Beteljuice

Saludos Beteljuice

First, I appreciate your passion for the games.

I have tested your utility to import cumulus data (meteobridge Pro red ) and, it worked.

My source file is in a metric format with a decimal sign (,).

It becomes your page well, but it loses the decimal value; for example: 9.6 becomes: 9

To solve this, I have made a previous change of the (,) by (.)

After this, the conversion has been done perfectly on your page.
Meteohub has imported all the data, without any error message.

The only slight inconvenience that I have been able to observe is that the rainfall data, on the days that it rained in abundance, are not exact; they are undervalued. But I must clarify that in your file, these data are correct; I think it is due to a problem of meteobridge itself.

- Javier
« Last Edit: May 13, 2019, 01:06:51 PM by kocher »
Windows 7 Pr
Davis Vantage Pro Plus f.a.
WeatherLink 6.0.2
Meteotemplate 15.0 Black Currant
CumulusMX
GraphWeather Version 3.0.15


Offline the beteljuice

  • the beteljuice
  • Contributor
  • ***
  • Posts: 128
    • test site
Re: Cumulus Data Import
« Reply #35 on: May 13, 2019, 08:22:10 AM »
FIXED:

Decimals 'lost' on format conversion and comma (,) decimal not being correctly dealt with in o/p file.

... both the same problem. the beteljuice had a slightly mistyped var name in an important place  :oops:

Naughty beteljuice  #-o

Quote from: kocher
I have tested your utility to import cumulus data (meteobridge Pro red ) and, it worked.

Clever betejuice  \:D/

Please try again .....

redone converter (same link)
Imagine what you will KNOW tomorrow !

Offline woolfg

  • Member
  • *
  • Posts: 26
Re: Cumulus Data Import
« Reply #36 on: May 13, 2019, 11:55:29 AM »
Hi beteljuice

I have tried your converter without success

I have attached a sample of my data (unfortunately I have cumulus data going back about 9 years which I would love to convert)

Can you have a look at my data and see if you can spot what is wrong

Kind regards

Graham

Offline kocher

  • Senior Member
  • **
  • Posts: 79
    • San Sebastian (Spain)
Re: Cumulus Data Import
« Reply #37 on: May 13, 2019, 12:36:44 PM »
FIXED:

Decimals 'lost' on format conversion and comma (,) decimal not being correctly dealt with in o/p file.

... both the same problem. the beteljuice had a slightly mistyped var name in an important place  :oops:

Naughty beteljuice  #-o

Quote from: kocher
I have tested your utility to import cumulus data (meteobridge Pro red ) and, it worked.

Clever betejuice  \:D/

Please try again .....

redone converter (same link)


Now it works perfectly.

Original file created by cumulusMX:

Code: [Select]
01/01/19;00:00;5,3;93;4,3;1,6;4,8;154;0,0;0,0;1033,39;1478,2;20,3;54;0,0;5,3;5,3;0,0;0;0,00;793,37;3,8;0;0,0;154;0,0;0,0
01/01/19;00:05;5,3;93;4,3;0,0;3,2;154;0,0;0,0;1033,36;0,0;20,3;54;0,0;5,3;5,3;0,0;0;-793,37;0,00;4,1;0;0,0;154;0,0;0,0
01/01/19;00:10;5,4;93;4,4;0,0;3,2;154;0,0;0,0;1033,42;0,0;20,3;54;0,0;5,4;5,4;0,0;0;-793,37;0,00;4,2;0;0,0;154;0,0;0,0

The same file, without any modification on my part, converted by Beteljuice:

Code: [Select]
"2019-01-01 00:00:00","5.3","93","4.3","1.6","4.8","154","0.0","0.0","1033.39","1478.2","20.3","54","0.0","5.3","5.3","0.0","0","0.00","793.37","3.8","0","0.0","154","0.0","0.0","N","SE"
"2019-01-01 00:05:00","5.3","93","4.3","0.0","3.2","154","0.0","0.0","1033.36","0.0","20.3","54","0.0","5.3","5.3","0.0","0","-793.37","0.00","4.1","0","0.0","154","0.0","0.0","N","SE"
"2019-01-01 00:10:00","5.4","93","4.4","0.0","3.2","154","0.0","0.0","1033.42","0.0","20.3","54","0.0","5.4","5.4","0.0","0","-793.37","0.00","4.2","0","0.0","154","0.0","0.0","N","SE"

You just have to put the cumulus file as it was created at the time.
Then the type of units we want are chosen.
we activate "debug" and ... Perfect!

meteobridge imports it without any problem

Betetjuice is a phenomenon!    =D> =D> =D>


- Javier
« Last Edit: May 13, 2019, 01:22:46 PM by kocher »
Windows 7 Pr
Davis Vantage Pro Plus f.a.
WeatherLink 6.0.2
Meteotemplate 15.0 Black Currant
CumulusMX
GraphWeather Version 3.0.15


Offline the beteljuice

  • the beteljuice
  • Contributor
  • ***
  • Posts: 128
    • test site
Re: Cumulus Data Import
« Reply #38 on: May 13, 2019, 02:07:24 PM »
@woolfg

That line appears to be in MB import format but without the " and last two compass points missing.

Is that your source file or the converter o/p ?

The converter would NOT recognise that as Cumulus data.

My test data was the few lines you posted earlier in the thread, and that works OK ....
Code: [Select]
01/09/18,09:00,16.7,90,15.1,0.2,2.6,99,0.0,0.0,1025.18,122.4,20.7,67,0.0,16.7,16.7,11.0,218,2.03,544.86,18.3,317,0.4,0,0.0,0.0
01/09/18,09:05,16.8,90,15.1,0.5,3.5,68,0.0,0.0,1025.25,122.4,20.7,68,3.5,16.8,16.8,11.0,217,0.10,544.96,18.2,331,0.4,67,0.0,0.0
01/09/18,09:10,16.9,89,15.1,0.9,3.5,68,0.0,0.0,1025.28,122.4,20.8,69,2.6,16.9,16.9,10.9,206,0.10,544.96,18.3,345,0.4,67,0.0,0.0
01/09/18,09:15,17.1,89,15.2,0.9,5.2,68,0.0,0.0,1025.35,122.4,20.8,69,5.2,17.1,17.1,10.9,152,0.10,544.96,18.4,359,0.4,67,0.0,0.0
01/09/18,09:30,17.7,88,15.7,0.9,3.5,73,0.0,0.0,1025.52,122.4,20.9,70,0.9,17.7,17.7,10.8,167,0.10,544.96,19.2,401,0.4,48,0.0,0.0
01/09/18,09:45,18.4,87,16.2,1.3,5.2,97,0.0,0.0,1025.58,122.4,21.2,70,0.9,18.4,18.4,10.7,250,0.10,544.96,20.0,441,0.4,92,0.0,0.0
01/09/18,10:00,19.3,85,16.7,1.3,3.5,178,0.0,0.0,1025.58,122.4,21.8,69,1.7,19.3,19.3,10.6,283,0.10,544.96,21.1,479,0.4,184,0.0,0.0
01/09/18,10:15,19.6,83,16.6,1.4,3.5,158,0.0,0.0,1025.62,122.4,22.3,68,0.9,19.6,19.6,10.7,309,0.23,545.08,21.3,516,0.4,240,0.0,0.0
01/09/18,10:30,19.9,80,16.3,1.0,4.3,167,0.0,0.0,1025.62,122.4,22.6,66,0.9,19.9,19.9,10.7,327,0.23,545.08,21.7,549,0.4,108,0.0,0.0
01/09/18,10:45,20.1,79,16.3,1.2,6.1,163,0.0,0.0,1025.72,122.4,22.9,65,0.9,20.1,20.1,10.9,341,0.23,545.08,21.8,581,0.4,117,0.0,0.0
Which converts to ..
Code: [Select]
"2018-09-01 09:00:00","16.7","90","15.1","0.3","3.8","99","0.0","0.0","1025.18","122.4","20.7","67","0.0","16.7","16.7","11.0","218","2.03","544.86","18.3","317","0.4","0","0.0","0.0","N","N"
"2018-09-01 09:05:00","16.8","90","15.1","0.7","5.1","68","0.0","0.0","1025.25","122.4","20.7","68","5.1","16.8","16.8","11.0","217","0.10","544.96","18.2","331","0.4","67","0.0","0.0","N","NE"
« Last Edit: May 13, 2019, 04:46:35 PM by the beteljuice »
Imagine what you will KNOW tomorrow !

Offline the beteljuice

  • the beteljuice
  • Contributor
  • ***
  • Posts: 128
    • test site
Re: Cumulus Data Import
« Reply #39 on: May 14, 2019, 09:24:22 AM »
Converter will now handle Cumulus month log data from ver 1.8.4 upto mx

Missing data is replaced by 0.0 or 0 or "N" as appropriate.

@woolfg ......

What's happening ? - It was your plight that made the beteljuice do this in the first place ?
Imagine what you will KNOW tomorrow !

Offline woolfg

  • Member
  • *
  • Posts: 26
Re: Cumulus Data Import
« Reply #40 on: May 14, 2019, 09:38:12 AM »
Converter will now handle Cumulus month log data from ver 1.8.4 upto mx

Missing data is replaced by 0.0 or 0 or "N" as appropriate.

@woolfg ......

What's happening ? - It was your plight that made the beteljuice do this in the first place ?

Hi

I am slowly getting my data imported . I have had a load of help from Lightmaster who looks after the Weather34 templates so between the both of you I am getting there. I just wish it wouldn't take so long to import each file

Thanks for your work too

Graham

Offline the beteljuice

  • the beteljuice
  • Contributor
  • ***
  • Posts: 128
    • test site
Re: Cumulus Data Import
« Reply #41 on: May 14, 2019, 05:41:43 PM »
New, improved, blah, blah, blah.

Now: Multiple file selection allowed - ie. marking a year of logs will produce ONE large 'import' file.

Cu > MB import converter

Imagine what you will KNOW tomorrow !

Offline lightmaster

  • MB-HWS Template Dev
  • Forecaster
  • *****
  • Posts: 302
    • Baxley,GA Weather
Re: Cumulus Data Import
« Reply #42 on: May 15, 2019, 04:54:52 AM »
This has kinda become more of a "Meteobridge Pro/NanoSD import issues" thread than a specifically cumulus one, but whatever, lol.

Found out from helping woolfg and txweather, that you must not have any duplicate sensors per timestamp, and timestamps must be in chronological order. With a Cumulus monthly file, you won't have duplicate sensors per timestamp unless you have 2 identical timestamps, however with a direct file which is a bunch of rows of "epoch-timestamp sensor data", you can accidentally create duplicate sensors.

If your getting an error about timestamps, and you know the timestamp is in the correct format for your import type, look above and below the row the system log complains about to see if it's all in the right order.

Also, if manually fixing files, DO NOT OPEN IN EXCEL unless you really know what you're doing. Excel likes to screw with dates and times and will change your timestamps. I've become very fond of VS Code, it has no issue opening .CSV and .TXT files with tens of millions of rows. (Excel maxes out a little over 1million rows).

Offline the beteljuice

  • the beteljuice
  • Contributor
  • ***
  • Posts: 128
    • test site
Re: Cumulus Data Import
« Reply #43 on: May 15, 2019, 08:38:04 AM »
Quote
If your getting an error about timestamps, and you know the timestamp is in the correct format for your import type, look above and below the row the system log complains about to see if it's all in the right order.
So ... are you saying if multiple Cumulus logs have been stitched together into one file and eg. Jan follows May everything will fold ?
Imagine what you will KNOW tomorrow !

Offline txweather.org

  • Forecaster
  • *****
  • Posts: 1488
    • Texas Weather
Re: Cumulus Data Import
« Reply #44 on: May 15, 2019, 11:05:44 AM »
the beteljuice, Can you give an option to import a raw file in imperial and export as a direct file in metric?

Even though Boris said it works it actually does not, lightmaster found that the files have to be in metric regardless.
And light has been awesome to help me convert but the problem has turned out that sometimes the timestamp is out of order and without some sanity checking the files will not fully import so it becomes painfully to fix manually.

I posted above how my data looks in meteohub raw files.

Thanks for your help!

----
Davis Vantage Pro2 Plus +FARS|Meteobridge Nano SD|Meteohub|Meteobridge MR-3020|WU KTXSPRIN75/PWS JRARGWX75/CWOP EW2972/WBB TXWDVUE75/Blitzortung ID: 1142|AWEKAS: 12095
Donations are welcome: https://paypal.me/ffuentesb

Offline lightmaster

  • MB-HWS Template Dev
  • Forecaster
  • *****
  • Posts: 302
    • Baxley,GA Weather
Re: Cumulus Data Import
« Reply #45 on: May 15, 2019, 11:48:34 AM »
Seems that there is a limit on how many records can be imported at a single time. Woolfg's yearly files stop importing on Oct 30th, but importing those last few months by themselves works.

Tx, if you're referring to your raw files, those are in metric, just have a decimal place without the actually period, ie: 0248 is 24.8C some other values however are integer values like the wind direction. The thorough documentation explains what it all is.


To summarize, so far, the rules are chronological order only, no duplicate sensors per timestamp, an unknown limit to how many records will import at once, and metric only (one of the formats has a value for if it's imperial, so that one may not have to be metric).

Also, I personally prefer the direct format because you can specify exactly which sensors the data goes to if you have multiple sensors (easily import data to the Pro's internal sensor).

Offline txweather.org

  • Forecaster
  • *****
  • Posts: 1488
    • Texas Weather
Re: Cumulus Data Import
« Reply #46 on: May 15, 2019, 11:53:34 AM »
Seems that there is a limit on how many records can be imported at a single time. Woolfg's yearly files stop importing on Oct 30th, but importing those last few months by themselves works.

Tx, if you're referring to your raw files, those are in metric, just have a decimal place without the actually period, ie: 0248 is 24.8C some other values however are integer values like the wind direction. The thorough documentation explains what it all is.


To summarize, so far, the rules are chronological order only, no duplicate sensors per timestamp, an unknown limit to how many records will import at once, and metric only (one of the formats has a value for if it's imperial, so that one may not have to be metric).

Also, I personally prefer the direct format because you can specify exactly which sensors the data goes to if you have multiple sensors (easily import data to the Pro's internal sensor).

Thanks for the correction light! I am tired asf. been awake for 24hrs :(

----
Davis Vantage Pro2 Plus +FARS|Meteobridge Nano SD|Meteohub|Meteobridge MR-3020|WU KTXSPRIN75/PWS JRARGWX75/CWOP EW2972/WBB TXWDVUE75/Blitzortung ID: 1142|AWEKAS: 12095
Donations are welcome: https://paypal.me/ffuentesb

Offline the beteljuice

  • the beteljuice
  • Contributor
  • ***
  • Posts: 128
    • test site
Re: Cumulus Data Import
« Reply #47 on: May 15, 2019, 03:17:00 PM »
Quote from: txweather
the beteljuice, Can you give an option to import a raw file in imperial and export as a direct file in metric?

Even though Boris said it works it actually does not, lightmaster found that the files have to be in metric regardless.
The formatted o/p file is in metric (regardless ..) - you just tell the prog what your native units are.

What do you mean by 'direct file' - The I/P file should be a Cumulus month log and the o/p file is in MB format, complete with 'falsified' zero values for differences in field length in Cumulus over the years. plus the two cardinal points at the end are calculated.

.. or are you talking about a non-cumulus format ???

@lightmaster .. So it's safer to just import one month at a time ?

Edit: beteljuice Cumulus converter modified - multiple files will be kept separate in one 'zip' file.
« Last Edit: May 15, 2019, 05:59:25 PM by the beteljuice »
Imagine what you will KNOW tomorrow !

Offline the beteljuice

  • the beteljuice
  • Contributor
  • ***
  • Posts: 128
    • test site
Re: Cumulus Data Import
« Reply #48 on: May 15, 2019, 09:44:05 PM »
Arggh !

After re-reading the MB site tutorial more closely realised I incorrectly set wind units to km/h instead of m/s  ](*,)

IMPORTANT - Anyone who has used the beteljuice converter should remove the data from MB and re-do again from the converter (now changed)

Sorry everyone - should have known it was going too well  :oops: #-o

Cu > MB import converter
« Last Edit: May 16, 2019, 08:18:10 PM by the beteljuice »
Imagine what you will KNOW tomorrow !

Offline lightmaster

  • MB-HWS Template Dev
  • Forecaster
  • *****
  • Posts: 302
    • Baxley,GA Weather
Re: Cumulus Data Import
« Reply #49 on: May 16, 2019, 06:36:01 AM »
Yes, monthly files seem to be best. You could probably get away with quarterly files, but they seem to take longer to import a few large files than it does to import many small files, probably due to limited system resources. I remember reading something that said all imported files are converted into direct files inside the system, then imported into the SQLite3 database. Presumably that happens in RAM which is limited, so a large yearly file would run out of RAM when it's converted into a direct file. The direct files are much larger than other files because the time stamp and sensor is written explicitly for each bit of data, so whereas a raw file would have 1 line with 1 timestamp, wind0, and 3 or 4 bits of data on a line, in a direct file that becomes 3 or 4 lines each with timestamp sensor/datatype, and then data. Meteohub Raw File to MB direct file results in roughly doubling the filesize. A cumulus monthly file with all the sensors on a single line, that's easily quadrupled in size.

So yes, smaller monthly files are much safer.

 

anything