Author Topic: CURL in Cumulus  (Read 7010 times)

0 Members and 1 Guest are viewing this topic.

Offline Jáchym

  • Meteotemplate Developer
  • Forecaster
  • *****
  • Posts: 8605
    • Meteotemplate
CURL in Cumulus
« on: May 15, 2015, 08:42:24 AM »
Hi,
I hope I am almost at the finishing line of putting together the first alpha version of my template. There is just one last - however absolutely essential - part which I am a bit unsure and dont now how to do. This is mostly due to the fact I am not familiar with any weather softwares, but still want to make my template compatible with these so I need to know how they work.

Basically what I need right now is:

I have a PHP script ready, which will take data, validate it and if they pass the validation process, saves them into the MySQL. However, I currently only have this script ready for use with Meteobridge, which I use myself and know how to use.

Meteobridge is able to run a php file, where it inserts dynamic tags into the url, so I am then able to get this data by PHP by the standard _GET command:
for example you can set the update interval to 1 minute and then every one minute, Meteobridge will run:
www.abcd.com/update.php?t=55&h=98..... etc.
Obviously by "opening" the url, the script is executed and does what it is intended to do.

With regards to Cumulus, this software currently does not have the function of regularly running a script (page). However, I have asked the developer and he suggested that there is some kind of a function of executing a defined program or code and that I could make a CURL job to do this. However I have never done any sort of programming of normal PC programs, only web. I looked at some documentation for CURL, but to be honest I wasnt able to figure out how to do this and especially how to set this up in cumulus. My idea then was that it would work like this:
1. Cumulus would upload the realtime.txt at regular intervals to the server
2. a PHP script on the server would contain a command to read that txt file and then work with the data as necessary

What I then need though is Cumulus not only updating realtime.txt (which is easily done), but also executing that script so that the realtime is read and data processed. I could make a CRON job to do this, but I would like to use this as the very last option if nothing else is possible, because I can imagine many people are unfamiliar with CRON jobs and how to set them up or dont even have that feature as part of their webhosting.

If anyone has any ideas and would be willing to help me with this I would very much appreciate it.

Thanks in advance

Offline PaulMy

  • Forecaster
  • *****
  • Posts: 5509
    • KomokaWeather
Re: CURL in Cumulus
« Reply #1 on: May 15, 2015, 03:15:23 PM »
Hi Jachym,
I am not very knowledgeable but do use the External Program feature in Cumulus to run a cmd program at my normal Cumulus update interval of 5 minutes (this cmd program written by someone else for another script I use).  I presume I can use the Realtime Program section if I wanted that to run at say my 5 sec realtime setting, or on a daily basis by using the Daily section.
The Help in Cumulus for external programs shows:
External Programs
ProgramThe name (including path) of a command-line program to be run at the 'normal' web update interval, after all of the processing has been performed. The corresponding 'params' box can be used to specify any parameters to be passed to the command.
Realtime ProgramThe name (including path) of a command-line program to be run at the 'realtime' web update interval, after all of the processing has been performed. The corresponding 'params' box can be used to specify any parameters to be passed to the command.
DailyThe name (including path) of a command-line program to be run once a day, after the daily 'rollover' has been performed. The corresponding 'params' box can be used to specify any parameters to be passed to the command.
I also update my MySQL Dayfile database once a day using Toolbox (add-on for Cumulus) with ImportCumulusFile.php.  Toolbox allows a file such as ImportCumulusFile.php to run on a specific time each Day or Hourly.
 
If I can be of any further help let me know.
Paul
« Last Edit: May 15, 2015, 03:17:00 PM by PaulMy »

Offline BCJKiwi

  • Forecaster
  • *****
  • Posts: 302
    • Silver Acorn Weather - N.Z.
Re: CURL in Cumulus
« Reply #2 on: May 15, 2015, 04:56:41 PM »
Jachym

I understood you were going to install Cumulus - have you done this?
I also suggested you read the Cumulus Wiki - I was not being facetious!

If I understand your current question correctly, what you are wanting to do is already a basic function of how Cumulus works - without the extra complications that you are considering.

If you read the Wiki and install Cumulus all will become clear (as it would for your same question re WD although the learning curve there will be much greater).

If you want to provide software for others you will need to provide them with instructions on how to set up their chosen software to work with yours. I do not understand how you feel able to do this if you have not installed, run and tested the functions and written the instructions without running that software.

If you install and run the software then you will have a clear understanding of how to set up any tag files required (if not already included in the software) and have them uploaded to the website (by the software).

Just to help you along,
1. Install Cumulus (even if you don't run it against your station)
2. Look in the Cumulus\web folder and at the template files there.
3. Look at the configuration/internet/files page and the wiki and help related to that page.

As someone who has developed and provided software for both Cumulus and WD I know there is no way to achieve a satisfactory result for others without running their target software.

Offline Jáchym

  • Meteotemplate Developer
  • Forecaster
  • *****
  • Posts: 8605
    • Meteotemplate
Re: CURL in Cumulus
« Reply #3 on: May 15, 2015, 06:54:37 PM »
Hi,
yes of course I have installed Cumulus before asking... but I was not able to figure out how that "running a program" function works. Otherwise I know cumulus pretty well in terms of how the generated file looks like and how to get the data and I have even tested my script already. The only thing I need is to make cumulus to run that script on a regular basis, which was the only thing I was asking.

Offline BCJKiwi

  • Forecaster
  • *****
  • Posts: 302
    • Silver Acorn Weather - N.Z.
Re: CURL in Cumulus
« Reply #4 on: May 15, 2015, 08:25:23 PM »
If you mean by "running a program" in Cumulus that you are trying to use the "Configuration/Internet Settings/Sites Options/External Progams" feature, then the there are a couple of ways to do that.

The one I use is to create a batch (.cmd) file for each of the three times available.
"Program" I call tagtime.cmd
"Realtime Program" I call realtime.cmd
"Daily" I call daily.cmd
So the Program box contains
    C:\cumulus\ftp\tagtime.cmd
    where C:\cumulus\ftp\ is the folder in which the batch files are located. The Params field is empty.
In the .cmd file(s) you can create any commands to run anything you need.

While on the "Site Options" page, click Help (button at bottom right corner of window) and drill down to configuration internet and scroll down to External programs (about 80% of the way down) which details the syntax and timing.

If this is not what you are trying to do, more detail on exactly what you are trying to do (in Cumulus terms) would help in getting you the right info.

If you need other timing then you will need to use a 3rd party program like a scheduler or Cumulus toolbox to set up the tasks.

One of the reasons I use batch files is that they are easy to test and debug directly without the involving the "external programs" scheduler in Cumulus. Once you know they work, then you can verify the "external programs" function is doing its job.

I find this a more flexible method rather than the program / Paramaters method.

The batch file also allows any number of operations at each interval.

However, if you are just trying to get tag files to the website, then you don't need any of this, just set them up in "Internet settings/Files". If there are not enough slots, you can add more directly in cumulus.ini.

Trust this helps.
« Last Edit: May 15, 2015, 08:36:35 PM by BCJKiwi »

Offline swyman18

  • Senior Member
  • **
  • Posts: 58
Re: CURL in Cumulus
« Reply #5 on: May 15, 2015, 08:51:30 PM »
Hi,

It sounds like the challenge is that the user needs to be able be able to execute the php script, regardless of whether it is using the built in Cumulus scheduler or something else. Obviously you don't want to assume that the standard Windows user has PHP installed locally. A Linux user using the CumulusMX beta can probably execute the script using the wget command. I think there is a wget utility for Windows, right?  Not sure how reliable it is.

It sounds like CURL might be a good option too, but unfortunately I don't know anything about it. Sorry.

For me personally, I have no problem running a wget cron job from my Ubuntu box but I know that is not really an option for a lot of people.


Offline Jáchym

  • Meteotemplate Developer
  • Forecaster
  • *****
  • Posts: 8605
    • Meteotemplate
Re: CURL in Cumulus
« Reply #6 on: May 15, 2015, 10:18:56 PM »
If you mean by "running a program" in Cumulus that you are trying to use the "Configuration/Internet Settings/Sites Options/External Progams" feature, then the there are a couple of ways to do that.

The one I use is to create a batch (.cmd) file for each of the three times available.
"Program" I call tagtime.cmd
"Realtime Program" I call realtime.cmd
"Daily" I call daily.cmd
So the Program box contains
    C:\cumulus\ftp\tagtime.cmd
    where C:\cumulus\ftp\ is the folder in which the batch files are located. The Params field is empty.
In the .cmd file(s) you can create any commands to run anything you need.

While on the "Site Options" page, click Help (button at bottom right corner of window) and drill down to configuration internet and scroll down to External programs (about 80% of the way down) which details the syntax and timing.

If this is not what you are trying to do, more detail on exactly what you are trying to do (in Cumulus terms) would help in getting you the right info.

If you need other timing then you will need to use a 3rd party program like a scheduler or Cumulus toolbox to set up the tasks.

One of the reasons I use batch files is that they are easy to test and debug directly without the involving the "external programs" scheduler in Cumulus. Once you know they work, then you can verify the "external programs" function is doing its job.

I find this a more flexible method rather than the program / Paramaters method.

The batch file also allows any number of operations at each interval.

However, if you are just trying to get tag files to the website, then you don't need any of this, just set them up in "Internet settings/Files". If there are not enough slots, you can add more directly in cumulus.ini.

Trust this helps.

Hi
Thanks a lot I will look into it.

I have also found a way of doing it with a vbs script which can be run on any Windows machine in the background

Offline BCJKiwi

  • Forecaster
  • *****
  • Posts: 302
    • Silver Acorn Weather - N.Z.
Re: CURL in Cumulus
« Reply #7 on: May 15, 2015, 11:56:34 PM »
Have recently started using vbs scripts to call the batch file for CumulusMX. The "extra programs" run in the foreground in CumulusMX (still in beta) and vbs can make them happen in the background.

@ Swyman18
There is no need to run PHP in windows with Cumulus.
Cumulus has a an engine which takes "tag" files and substitutes the "tags" with current values and the files are then FTP'd to the website. The website then processes the files (using PHP or .js or whatever the website is set up to use) to provide the pages for the Browser.

Jachym is taking the extra step of placing this data into SQL at the server. There are existing (standardised processes being used by many as an add-on to Cumulus) ways to get Cumulus data into SQL but Jachym appears to be doing this his way to suit his software.
« Last Edit: May 15, 2015, 11:59:13 PM by BCJKiwi »

Offline swyman18

  • Senior Member
  • **
  • Posts: 58
Re: CURL in Cumulus
« Reply #8 on: May 16, 2015, 12:27:45 AM »
Have recently started using vbs scripts to call the batch file for CumulusMX. The "extra programs" run in the foreground in CumulusMX (still in beta) and vbs can make them happen in the background.

@ Swyman18
There is no need to run PHP in windows with Cumulus.
Cumulus has a an engine which takes "tag" files and substitutes the "tags" with current values and the files are then FTP'd to the website. The website then processes the files (using PHP or .js or whatever the website is set up to use) to provide the pages for the Browser.

Jachym is taking the extra step of placing this data into SQL at the server. There are existing (standardised processes being used by many as an add-on to Cumulus) ways to get Cumulus data into SQL but Jachym appears to be doing this his way to suit his software.

Understood, thanks.  I am aware of how the webtags work, I thought he was trying to find a way to have the user execute the PHP script URL from a Windows machine which would load the data from the realtime.txt file (or similar) into a SQL database (local or remote).

Regardless, sounds like he's got it all well in hand.

Offline BCJKiwi

  • Forecaster
  • *****
  • Posts: 302
    • Silver Acorn Weather - N.Z.
Re: CURL in Cumulus
« Reply #9 on: May 16, 2015, 12:39:19 AM »
You may well be correct, I am having difficulty working out exactly what he is trying to do.

I guess my point is that there are well proven methods may use, just not sure if Jachym is trying to work out how to use them or do something different.

Offline Jáchym

  • Meteotemplate Developer
  • Forecaster
  • *****
  • Posts: 8605
    • Meteotemplate
Re: CURL in Cumulus
« Reply #10 on: May 16, 2015, 10:31:27 AM »
Hi,

thank you all for your tips and I also appologize because I probably didnt explain it really well. I will try again.

So basically with regards to getting data from station, there will be two stages in my template.

Stage 1: Loading history data into the database so that the user doesnt have to start from scratch (database etc. is created also automatically, I made scripts for that too)
- I have this part finished, I made 3 scripts, one is able to read, process and upload data from WU site directly, one does the same thing and is built to reflect the structure of WD log files and the last one again does the same thing, but with the monthly logs from Cumulus. This part is fine.

Stage 2: Continuous upload of current conditions
- so once stage 1 (which is optional, it is of course possible to start with a blank database) is finished, the goal is then to continuously add new data as they are measured and add them to the database.

The overall process can be summarized like this:
PWS -> software -> PHP script -> MySQL

PWS -> software - this is not an issue and can be done by various ways depending on which software the user uses
PHP script -> MySQL - again this is not a problem now, I have already made a script that is able to write the data into the db including data verification

The problematic part is the software ->PHP path.

The exact way of how to do this again depends on the particular software. For now I wanted to concentrate on three different: Meteobridge and similar, Cumulus and Weather display.

Meteobridge: this is probably the easiest. Meteobridge is capable of making an HTTP request, which means you put in the address of your script and it adds the current conditions into the address and can be then used by the script:
Example:
You set your meteobridge to send an HTTP request every 5 minutes to www.abcd.com/update.php?t="tag for temperature"&h="tag for humidity".....
Meteobridge then every five minutes sends request to for example www.abcd.com/update.php?t=55&h=23..... and the PHP script gets the data via the GET method.

And now we finally come to the problematic part. Cumulus and WD probably cannot do HTTP request like this, or maybe WD can in the newest version, but I will have to further look into it.

I therefore made a different script for these two softwares, which instead of getting the current data through the GET method and URL, gets the data from the txt file - i.e. realtime.txt or clientraw.txt. This again is not so difficult and I already have the code for this which works.

The final issue now is that obviously for the PHP script to load the text files and save the values to MySQL, I need the script to be executed at regular intervals - and this is what I am having the difficulty with.
CRON job - probably the easiest, but also problematic. First of all I can imagine lot of people dont know what it is and how to set it up. Second, not all webhostings have this feature and third, the interval offered by most webhostings is only very rarely as short as 5 minutes in the cheaper plans. I would therefore like to try to avoid CRON.

My question in this thread was therefore if anyone has an idea what would work the best and also what would be the most user-friendly option.

UPDATE:
As I said previously, I am not a PC programmer, I only know languages like PHP, Ajax, Javascript etc. but not C, Java or others. Still, I tried to look at some examples and I put together a Visual Basic VBS script, which at a user defined interval executes the PHP script. This seems as an ideal option, because it is not dependent on any weather software and can be run on any Windows computer.
My idea right now is therefore following (but again Im not sure to what extent it is feasible or if I maybe overlooked something)

Set the built-in windows task-scheduler to open and run this VBS script automatically upon opening whatever weather software you use. So for example set task scheduler to run this script everytime Cumulus is active. This script that contains the command to execute the php script (without the need to open the browser and performed in the background). This command is inside a loop so it is repeated over and over until the VBS script is running (which would be basically all the time if you have your weather software and data sent 24/7.) Also, between each loop there is a command for pausing (sleep), so you set this to for example 5 mintues and what you get is the PHP script then being executed every 5 minutes. I checked also the processes in windows and the resources and memory used by the VBS are negligible (about 3MB). So my idea right now is to use this VBS script because as I said, it can be used on any Windows machine irrespective of the weather software used and Im sure it will not be a problem to make something similar for iOS if necessary. And this would then be used by those who do not have a software that could make HTTP request directly.

Do you think this would work?

Offline Jáchym

  • Meteotemplate Developer
  • Forecaster
  • *****
  • Posts: 8605
    • Meteotemplate
Re: CURL in Cumulus
« Reply #11 on: May 16, 2015, 10:43:58 AM »
Or maybe I thought if WD or Cumulus (or any other software) would be able to run a VBS script directly at regular intervals, then I could delete that sleep command in the script, have it just execute the PHP once and then have the weather software to run this script automatically without even a need to set up task schedule

Offline PaulMy

  • Forecaster
  • *****
  • Posts: 5509
    • KomokaWeather
Re: CURL in Cumulus
« Reply #12 on: May 16, 2015, 12:42:08 PM »
Hi Jachym,
I understand bits and pieces of what you are working on, but a lot I don't understand.  Now this is not an issue if you have the template set working and all I, or any user, would need to do is install, setting, and run it.  And as I understand, correct me if I am wrong, that with your template set the weather station data produced by the software, i.e. Cumulus, is saved in a new my MySQL database.  You then use this data from the MySQL database to display within the template script.  The first step is the one-time optional step to get any historical data from the software in the MySQL database.  This data would be at the interval that the historical data had been saved by the software, i.e. 10 minutes, in the log file(s).  The next step is to get ongoing live data from the software into the MySQL database at the realtime interval, i.e. 5 seconds, or at the normal update interval, i.e. 5 minutes, and this is your current issue you are trying to resolve. 
 
If my understanding is correct so far, then how is the past data added to the MySQL database for the time period that the software is not running (i.e. Cumulus shut down as some people don't run 24-7 or when computer maintenance/update is being done)?  This is the "Catch Up" feature currently in the software to bring uploads to WU, PWS, WOW, etc. up to date.
 
Paul
 
 

Offline swyman18

  • Senior Member
  • **
  • Posts: 58
Re: CURL in Cumulus
« Reply #13 on: May 16, 2015, 12:48:57 PM »

Or maybe I thought if WD or Cumulus (or any other software) would be able to run a VBS script directly at regular intervals, then I could delete that sleep command in the script, have it just execute the PHP once and then have the weather software to run this script automatically without even a need to set up task schedule

It sounds like this should work, assuming the vbs script can be put into a batch file. Then cumulus could trigger the batch file at regular intervals.

You way also want to make the PHP script available for users who choose to execute via cron job. In my case, that is what I would use since I am running CumisMX (beta) on a Linux server. I'm assuming VBS is windows only, correct? I certainly could be wrong.

Offline Jáchym

  • Meteotemplate Developer
  • Forecaster
  • *****
  • Posts: 8605
    • Meteotemplate
Re: CURL in Cumulus
« Reply #14 on: May 16, 2015, 01:25:03 PM »

Or maybe I thought if WD or Cumulus (or any other software) would be able to run a VBS script directly at regular intervals, then I could delete that sleep command in the script, have it just execute the PHP once and then have the weather software to run this script automatically without even a need to set up task schedule

It sounds like this should work, assuming the vbs script can be put into a batch file. Then cumulus could trigger the batch file at regular intervals.

You way also want to make the PHP script available for users who choose to execute via cron job. In my case, that is what I would use since I am running CumisMX (beta) on a Linux server. I'm assuming VBS is windows only, correct? I certainly could be wrong.

Hi, well that VBS script would only execute the PHP script so in this case you would still use the same PHP script, just instead of triggering it by VBS, you would trigger it by CRON.

Offline Jáchym

  • Meteotemplate Developer
  • Forecaster
  • *****
  • Posts: 8605
    • Meteotemplate
Re: CURL in Cumulus
« Reply #15 on: May 16, 2015, 01:33:14 PM »
Hi Jachym,
I understand bits and pieces of what you are working on, but a lot I don't understand.  Now this is not an issue if you have the template set working and all I, or any user, would need to do is install, setting, and run it.  And as I understand, correct me if I am wrong, that with your template set the weather station data produced by the software, i.e. Cumulus, is saved in a new my MySQL database.  You then use this data from the MySQL database to display within the template script.  The first step is the one-time optional step to get any historical data from the software in the MySQL database.  This data would be at the interval that the historical data had been saved by the software, i.e. 10 minutes, in the log file(s).  The next step is to get ongoing live data from the software into the MySQL database at the realtime interval, i.e. 5 seconds, or at the normal update interval, i.e. 5 minutes, and this is your current issue you are trying to resolve. 
 
If my understanding is correct so far, then how is the past data added to the MySQL database for the time period that the software is not running (i.e. Cumulus shut down as some people don't run 24-7 or when computer maintenance/update is being done)?  This is the "Catch Up" feature currently in the software to bring uploads to WU, PWS, WOW, etc. up to date.
 
Paul

Hi Paul,
yes you understand it absolutely correct, first you upload historical data if you want and then you continue with uploading new "rows" as they are measured. The only condition is that the intervals should be the same. My recommendation is using 5 minutes. Having longer is not ideal, but also not a problem. If your historical data is in 10min intervals, you would copy each record twice, just to make sure the averages are calculated properly.
Also I have implemented measures that allow using 5min intervals and still not get biased results. I will explain that in the documentation, but basically it means that measuring in for example 1min would be useless and this is also to make it much faster in terms of calculations. Keep in mind it is all calculated in realtime so having a huge database with redundant data is not something you want :-)

Now, to your question. If I understand it correctly, what you mean is how it would deal with people who dont have Cumulus running 24/7 (or other SW). Well I never really thought about this, but in fact it will not be a problem either. Because... basically what you would do is:
1. you would initially upload the entire historical data
2. then you would keep on adding new data as it comes.
If you then had your PC off, your data will be downloaded retrospectively and saved in the log file. So I guess you would then go to the same script that imported the historical data and upload that log file which contains the data you want. Each data set has a timestamp, which is saved in the database so this means that if there is some data missing, there simply wont be any values for that time. And also, once you upload a new file, it will add the new rows with their time and date into the database and if some value is there already, it will simply overwrite it with the same value, so it doesnt matter. There is a primary key set on the timestamp to make sure there is never a duplicate value for the same data set - if same date/time is encountered, it overwrites it, thus not causing problems with duplicates. This can also be used to correct data. You just change it in your log file and upload it and the changed values are changed in the DB.

So as far as I can see this should not be a problem.. hopefully :D

Offline Jáchym

  • Meteotemplate Developer
  • Forecaster
  • *****
  • Posts: 8605
    • Meteotemplate
Re: CURL in Cumulus
« Reply #16 on: May 16, 2015, 01:39:25 PM »
One last thing .....

there is something that I dont really understand and that is - all these softwares offer built-in function to upload to for example WU. Now as far as I know, the WU upload is done by an HTTP request - which is exactly what I need....
So I assume all these programs in fact can perform HTTP request, just dont have the possibility of the user defining custom URL and only use it for the predefined WU URL.... but in that case, creating a field and option to set own URL should be very simple and easy, because that function in fact is already there. But maybe I am missing something, I am not a PC programmer :D

Offline PaulMy

  • Forecaster
  • *****
  • Posts: 5509
    • KomokaWeather
Re: CURL in Cumulus
« Reply #17 on: May 16, 2015, 03:04:47 PM »
Hi Jachym,
Quote
One last thing .....

there is something that I dont really understand and that is - all these softwares offer built-in function to upload to for example WU. Now as far as I know, the WU upload is done by an HTTP request - which is exactly what I need....
So I assume all these programs in fact can perform HTTP request, just dont have the possibility of the user defining custom URL and only use it for the predefined WU URL.... but in that case, creating a field and option to set own URL should be very simple and easy, because that function in fact is already there. But maybe I am missing something, I am not a PC programmer :D
Other than that Cumulus can do the "Catch Up" to the several sites using the protocol specified by each site, I don't know how that works.
Quote
So I guess you would then go to the same script that imported the historical data and upload that log file which contains the data you want.
As I run Cumulus 24/7 that would usually not be an issue for me but for those that don't run the software or their computer all the time, and I think there are quite a number like that, it would be a hassle to re-upload every time.  But there are occasions when I shut down my computer for updates, etc. and Cumulus knows when the last station/logger download was made so it catches up automatically at restart.  I presume your though is that the current month's log file be re-uploaded in full in thos circumstances and any duplicates would be ignored.  It is also been a convenient feature that when there was corrupt data one could just rewind by copying in the data files from the previous day's automatic Cumulus backup, or any of the past 10 days backups, and paste in the current folder and then restarting Cumulus which would then download from the logger from that backup date, being a very quick and reliable restore.  I presume your thought is that the same one-time upload of the log file would be done at those times as well?
 
You mention that the interval times should be the same.  Unfortunately my historical records have both 30 minutes interval  when I first started the weather station in 2008, changed to 15 minutes after the first year or so, and now for the past several years at 10 minutes.  My data logger has been set at the same interval times.  However my web upload is set at 5 minutes.  Will that be a problem? 

Paul
 
 
 

Offline Jáchym

  • Meteotemplate Developer
  • Forecaster
  • *****
  • Posts: 8605
    • Meteotemplate
Re: CURL in Cumulus
« Reply #18 on: May 16, 2015, 06:48:42 PM »
Hi Paul,

on one hand it does mean that if you turn your computer off you obviously will have to upload missing data retrospectively, on the other hand, the upload only takes a few seconds and there really is no othre way of getting the missed information to the db.

With regards to your measurements, it is a slight problem if the intervals are inconsistent, but not a problem that could not be solved. As I mentioned earlier, you would then need to duplicate the measurements to reflect the new interval.

Example:
Initial measurements in 10min interval:
2:20 T=55 H=90
2:30 T=56 H=88
2:40 T=58 H=85
New measurements
8:30 T=75 H=60
8:35 T=75 H=61

Obviously the problem would be if you mixed these together because the average for example would be biased. What you could do however, which would solve the problem, is duplicate those values so that they would reflect the twice as long interval, i.e.:
2:20 T=55 H=90
2:25 T=55 H=90
2:30 T=56 H=88
2:35 T=56 H=88
2:40 T=58 H=85
2:45 T=58 H=85

Offline PaulMy

  • Forecaster
  • *****
  • Posts: 5509
    • KomokaWeather
Re: CURL in Cumulus
« Reply #19 on: May 16, 2015, 09:12:00 PM »
Thanks Jachym,
Just wanted to understand, and also to mention what I think is quite a common practise for your consideration in your template's development.

Paul

Offline Jáchym

  • Meteotemplate Developer
  • Forecaster
  • *****
  • Posts: 8605
    • Meteotemplate
Re: CURL in Cumulus
« Reply #20 on: May 16, 2015, 09:21:23 PM »
Sure,  thanks for mentioning it,  the only problem is that those people will always see the latest value in dB or txt file as current conditions

Offline BCJKiwi

  • Forecaster
  • *****
  • Posts: 302
    • Silver Acorn Weather - N.Z.
Re: CURL in Cumulus
« Reply #21 on: May 16, 2015, 10:00:37 PM »
Did you find this in the Cumulus WIKI?
http://wiki.sandaysoft.com/a/ImportCumulusFile
realtime.txt is also supported though missing from some of the documentation.
« Last Edit: May 17, 2015, 06:27:13 AM by BCJKiwi »

Offline mcrossley

  • Forecaster
  • *****
  • Posts: 1132
    • Wilmslow Astro
Re: CURL in Cumulus
« Reply #22 on: May 17, 2015, 05:29:15 AM »
WD does already have the ability to use a user defined http get, and according to Steve's reply to you on the Cumulus forum, CumulusMX will have the ability to remote upload to MySQL either directly or via http get in the next release.

I'm really not following the logic in your update process, you seem to be proposing scripts to call other processes, all kicked off by cron (or Windows scheduler), when both Cumulus and WD can invoke tasks directly themsleves when the data changes.

Also if you have a unique primary key on a MySQL tables, duplicate rows will not be overwritten by default, the insert will fail, the ImportCumulus script has been coded to silently ignore duplicates To avoid the overhead of reinserting and potential for fragmenting the table unnecessarlily.
Mark

Offline nitrx

  • Senior Contributor
  • ****
  • Posts: 277
    • Apeldoorn-Oost The Netherlands
Re: CURL in Cumulus
« Reply #23 on: May 17, 2015, 06:20:51 AM »
It's a bad idea to use Windows only based commands as Cumulus successor CumulusMX wil work on Unix and Mac too many users are running the beta of CumulusMX on a Rasperry Pi.
Ron
Cumulus software
Apeldoorn NL
http://www.apeldoornmeteo.nl

several languages

Offline Jáchym

  • Meteotemplate Developer
  • Forecaster
  • *****
  • Posts: 8605
    • Meteotemplate
Re: CURL in Cumulus
« Reply #24 on: May 17, 2015, 10:13:02 AM »
Hi,
yes, but remember, the actual script that will be executed is separate, which means that you can use any other way of trigerring it. This VBS script is only to run the other script at regular intervals, but it doesnt really matter which way you choose to do this. If somebody for example is familiar and have the possibility of doing it with a CRON job, then no problem. If somebody downloads the new Cumulus MX, which will be able to do it, again, it is ok. So this VBS would be only as a "back-up" for those that do not upgrade to Cumulus MX and for example dont have the possibility of setting up CRON job so often.

Quote
Also if you have a unique primary key on a MySQL tables, duplicate rows will not be overwritten by default, the insert will fail, the ImportCumulus script has been coded to silently ignore duplicates To avoid the overhead of reinserting and potential for fragmenting the table unnecessarlily.

Yes you are right, but I just didnt express myself correctly. The script actually would rewrite the data, but not because there is a primary key on date, but because there is a specific part in the code that checks whether that date is already in there and if so, updates that row.

Quote
Did you find this in the Cumulus WIKI?
http://wiki.sandaysoft.com/a/ImportCumulusFile
realtime.txt is also supported though missing from some of the documentation.

Yes, Ive seen this script, but I dont actually need it. The one I wrote is slightly different in the way it checks and validates data and subsequently saves them into MySQL and it works fine, the data is saved correctly, Ive checked that, so there is no need to change it, but thanks for the tip.