Topic: My new template

[Jáchym]

So as promised, finally after about of month of intensive work over nights I managed to put together first preview of the template I am working on. So far it is still a long way to go and I am only posting some screenshots and videos. There are two videos showing the actual template and one video showing how the settings will look like. It is just for you to get an idea of what I am doing. I also made a few higher-resolution screenshots of randomly generated design combinations, which I am attaching as a ZIP file.

if you are interested you can watch them and in hopefully near future I will have a first “beta release”, which will not be just a video, but the real thing and to those that showed interest in testing I will send a PM or email.
Just a couple of important things before watching:

1.   Most importantly… please excuse the video quality. The real thing is of course nice and sharp, but when I tried recording in HD, my 1GB RAM laptop completely froze… even the medium quality led to BSOD, so I had to record on the lowest quality. Still I hope in fullscreen most of the fonts are legible. Also the lagging and slow loading is due to the fact I was screen capturing while looking at it and it normally runs much faster.

2.   The template is currently hooked to my own PWS and database. I measure data since 1st January 2012, which is why in all cases you will see that data and menus etc. go back to 2012, but this of course will adjust depending on what your data will look like.

3.   Video number 3 is a preview of how the configuration will look like. One of the options is also solar sensor. My WH1080 doesn’t have one, which is why all the graphs and tables in the preview for solar radiation are blank. Normally if you don’t have one, you would set this parameter in the config file to false and all tabs, tables, icons etc. of the solar radiation will not be visible on your site. But for demonstration purposes I left it on true and therefore had zeros everywhere.

4.   The template uses purely javascript, CSS, PHP etc., but no potentially unsupported plugins such as Flash, Silverlight and also no cookies. Compatibility tested in Chrome, Firefox, newest IE, Opera, Maxthon, Nitro and also on mobile devices (Android). I don’t have access to a Mac, but tried it once in Safari on iPad and iMac and worked ok.

5.   Everything you see is calculated in real-time, which means even the most recent data is taken into account in all the graphs, stats etc. In order to achieve fast loading times I tried to implement AJAX and dynamic loading as much as possible and there are also mechanisms that for example pre-load other tabs in the background while already show you something (you will see what I mean if you watch it).
Keep in mind this is just the very first version and still long way to go. My plan is to have more sections. The videos only show the “weather station” section, then you will have options of many different “plug-ins” or pages that can be added. It is fully compatible with basically anything and I will then be releasing new scripts, which I will call “plugins” and just by placing them into the plug-in folder of the template, they will show up in the main menu. So for example there will definitely be my already released scripts – the forecasts, banners, climates, observations, I will also do many more based on my current personal webpage and new ones. So for example the top banner can be the one I already made, interactive one etc.

An essential part of the whole template will also be scripts to help you with the initial set-up – i.e. scripts that will automatically create databases and tables and I hope I will be able to also make scripts that will automatically import historical data from outputs from software such as weather display, cumulus or even fetch them from WU. I have already done some testing and it should be possible. I recommend using 5min intervals in the db, which meant 1 yr worth of data took about 30 s to import and remember you only do this once at the beginning, so if you have say 10 yrs of data, it would take you maybe 20 minutes to set up. If you measure at shorter intervals (1 min) it is of course possible to use as well, but 5 mins is ideal and there are mechanisms integrated that make sure you don’t miss any extremes etc. even with 5min interval, which I would explain later. Importing scripts will do everything automatically and I plan to also make checking mechanisms, that will for example check the data is valid, send alerts via emails etc. 

Yes I do have lot of plans

So in conclusion, this is really just the first preview of the first part, but I wanted to have some feedback already so that if there is something you think I should do differently or if you think this whole thing is not really worth finishing, let me know and I will take all comments into account.

Videos:
PART 1 - https://www.youtube.com/watch?v=iryjHSe3Nbk
PART 2 - https://www.youtube.com/watch?v=WEOmTe2DsQc
SETTINGS - https://www.youtube.com/watch?v=ahs6Gq305fw
Screenshots: http://www.megafileupload.com/1OkM/imgs.zip

[Farmtalk]

Looks VERY promising! Congratulations on a job well done! Looking forward to see how others react when it is released! 

[wxkpt]

Great  I know you have been working very hard on this. I cannot wait for the finished product. It looks like it's going to be a great addition to the weather community. Looking forward to the finished script.

[virusdunil]

A completely new desing of template...great job...cant wait 

[Jáchym]

Thanks, I think the first real demo you can test yourself should be ready within 1-2 weeks

[virusdunil]

 ill wait hehehe 

[n9mfk9]

can not wait to try it

[nitrx]

Looks great  count me in for the beta hope it works with Cumulus

[Jáchym]

Hi, thanks for your feedback, I will copy part of my post from the other forum, which answers your question in terms of Cumulus compatibility:

I think I should clarify a bit more the fundamental difference between all the current templates such as Ken´s or Leuven template. In a nutshell, this is how they work:

PWS sensors -> PWS console -> computer -> Weather Display / Cumulus / any other SW -> data analysis, graph generation, statistics etc -> generation of txt file with the calculated parameters -> text file with parameters (in case of WD called clientraw) -> webpage with dynamic fields -> fields filled in with data from text file -> data shown on webpage

And here is how my template works:

PWS sensors -> PWS console -> computer/or just a router such as meteobridge etc. -> just the several raw data parameters from sensors sent to my PHP scripts -> scripts check validity save to db -> webpages (i.e. my template) use raw data and data in created table to do all statistics, graphs, visualizations etc.

So, the major difference is that in the first case, the template is basially just a fixed page that plugs in values from whatever software you use. The advantage of such a solution is that obviously it is relatively easy to do, you just put those dynamic markers on the page and your weather software fills them in. In my case, you have to program all the scripts yourself so that they are actually the thing that calculates everything and shows it on the page.
And the reason why I chose this option and the major advantages are as follows: obvously, if you go for the first option, you will always be limited by whatever parameters and data are available from the SW. Even though WD generates quite extensive set of data, there are still many things that I for example didnt find there or you just want something particular and if it is not in the clientraw... The other huge advantage is that you can actually if you want not use any sw at all. This was in fact my case. I have a PWS connected to meteobridge. Meteobridge is a tiny router with minimum consumption of a size maybe 5x5 cm. It is a great thing because you do not have to (and in my case that would not even be possible) to have your computer on 24/7, it has a very low consupmtion, is absolutely quiet etc. The problem however is that obviously, it is small and has a very little memory and it cant do much more than just send the data "somewhere". So this way, you are quite limited in terms of what is available. However, in my case it is absolutely fine. All you need in my template are that several parameters directly measured by PWS. My scripts calculate all the stats, all the graphs, apparent temperature, moonphases etc. etc. Everything is done on the web, it also allows it to be fully interactive and expandable and in the future I can program anything I want.

I hope I explained how it actually works and the main differences. But of course this also means that you will still be able to use it with any SW, in fact the good thing is you will be able to use it with absolutely any SW, because even though WD probably has no competition in terms of the amount of data it provides, other SW might be more limited, but in this case since all you need is a few numbers for temperature, humidity, wind speed, direction, gust, pressure and maybe radiation if you have that sensor, all the other things are calculated, including apparent, dew point, all the stats, records, averages, historical values and that is then used to generate all tables, graphs etc.

[virusdunil]

So...if i understand right...as a WD user...i will have to put lets say %temp% in the template to have the actual temperature readings...right ?

or i completely lost it hehehe

[Jáchym]

Yes, you are completely lost

My template is completely different from all the others, it doesnt use WD, Cumulus or anything to do the calculations, graphs etc. for it. It does it by fetching data from a db, which it has. In terms of this database, I will make automatic scripts that will take care of this database. And these scripts will have several versions depending on how you want to get your data to the database, many options - couple examples: I will definately make one script that will use the clientraw from WD (it will only take the raw parameters from sensors from it, save it to db and all the rest will be done by the template scripts), one version for Cumulus, I am currently working on a script that will make the db from WU if you send data there, as a user all you will have to do is put your WU ID, one script for meteobridge directly (so no need to use any sw or computer, just meteobridge data) etc. etc.

[Jáchym]

So say you want to use the template, then there are three steps:

STEP 1:
Create db with a proper structure etc. - all done by an automatic script, the user just has to run the script once and thats it.

STEP 2 (optional):
If you do not want to start with a blank database and already have data from the past, then at the beginning you will have to import your existing data to the created DB. This is what I was talking about above - there will be several options of how to do it. I will make custom-made script for importing from WD log files, one for Cumulus, one for WU and possibly any other should ppl want it. This you only do once at the beginning, to have the db ready and it is optional. And even then, as I said, it will be very simple in terms of the user input - such as uploading log files to a specific folder or specifying WU ID etc. But all the programming stuff I will do myself using the scripts.

STEP 3:
Once you have your db, all you have to do is regularly send data to it. And here, again there will be many options, I will make one script that will regularly send data using clientraw, one for Cumulus, one for meteobridge etc. So in your defined interval, the db will be getting new set of variables.


And that is how it works, which is why it is so customizable, because the raw data can be imported from anywhere, not even a computer, just a simple router and everything including stats, graphs, tables etc. is then done later by the scripts and displayed by the template automatically.

[mcrossley]

So effectively your scripts become the weather software rather than Cumulus, WD, etc.

[Jáchym]

I was actually afraid you or Brian from WD are gonna say something at this point

But dont worry... there still will be many advantages of ppl using normal PC software, for example if your connection is down, you will be able to upload missed data reciprocally, but at that time, you wont have access to your data, as it will be in the database on the server. Another thing is that the template relies on the data being supplied to it from "somewhere" - Cumulus, WD etc. it cannot directly connect to your PWS obviously.... So I would still advise everyone to use your software or WD or else, but the advantage of this is that obviously cumulus or WD can only provide certain text files and universal data, but this allows anyone to do whatever stats or tables or graphs they want and present them in a way they want them to be presented.

Either way, I intend my template as a nice "add-on" and used by ppl in combination with any software they are already using. And even though I said you could theoretically do completely without any SW, for example in case of meteobridge, which can send data straight to MySQL, those people can already do it and dont need my template because they could be just sending data to for example WU and that way already not be using any SW. Either way, there will be no change, just an add on

And because I find particularly your and Brian´s WD the best, I will make sure those are the two that will definately be supported.

[mcrossley]

To be clear, I have nothing to do with Cumulus other than being a user, I too have implemented a MySQL dB for my Cumulus data in order to do things you cannot get from the standard program. I see what you are trying to do, but as you say if you want to make use of the logger features of the weather stations then you still need some software sitting locally to manage that.

When I first started on this hobby I had a dim hope that there would be a 'standard' for data interchange between programs, something probably XML or JSON based. I soon saw that wasn't the case, even among the 'professionals', every program and organisation seems to reinvent an incompatible wheel.

What I would like to see would be something like what the ASCOM or INDI people have done for astronomy drivers - heck they have even included some rudimentary weather data standards in their interface! That would make it far simpler for people to integrate the multitude of weather programs into standard templates thereby increasing choice.

Anyway, that is a diversion, thread back to your template...

[Jáchym]

Hi Mark,

in my case it was like this too... exactly like this.... I knew nothing about HTML, PHP or anything. I bought my WH1080, connected it with a USB to my computer. It was supplied with EasyWeather, which is a free program, which however cant do anything. I came across WD, but unfortunately it was something I cant afford, so I used the free Cumulus. Over time however, I realized I would like to do other things and also bought meteobridge, because I cant have my laptop 24/7. Until then I was always uploading data manually every evening, but I wanted real-time data.
And once I bought meteobridge, I started being interested in what actually is HTML and then I started reading about it and realized I wont be able to do much with just HTML because it is static, so I tried javascript, then Php and it went on and on, I came across ajax, css..... whatever, the point is, this was 2 yrs ago and I was also using Cumulus at the beginning but there were just things it cant do and that is why I needed something like my template, which would complement it.

[nitrx]

I stay optimistic wondering how you will handle al time and date and value issues 

[Jáchym]

What exactly do you mean "handle issues"? Sorry, I dont really know what you mean by that.

[nitrx]

Oh sorry Jachym I meant I've a comma decimal seperator in my data like 4.5 some have 4.4 dates are different in a lot of countries like dd-mm-yyyy yyyy-mm-dd andso-on but I'm sure you can handle that in your scripts

[staccermaccer]

Hi, Jachym

That is unbelievable that you managed those skills in 2 years. And the template looks awesome! Keep up the good work, i am still using your adapted weather-dependant banner. How did you learn so fast? I thought i was kind of smart, but you surpass me by miles... 

Now i am really an admirer of you....

Great job,

Marc,

Antwerp

[Jáchym]

Well it was more like a year, not two, because the first yr I was using Cumulus and not doing much... but I am saying that not to praise myself The thing is, I know I learn this really fast even though my free time is quite limited, last yr I graduated from Masters and now I have a full and part-time job and the reasons are probably these:

Couple of points that I found important:
1. you must enjoy doing it - this is the key, then everything goes much faster
2. set yourself both short- and long-term goals
3. dont read books etc. Just make a plan of what you want to do and then once you get to a point when you dont know what to do next, look on the web - here on the forum, stackoverflow etc. and you will find that 90% of the time someone has already had that problem.... - trial/error method is the best, learn from practice, not theory
4. then when you for example offer your work to someone else and they write you it helped them or you help them in some other way, it makes you feel really good and you know what you are doing is worth it and you stay motivated
5. learn to sleep quickly so that you have time to work on your scripts like me during night when normal people sleep
6. dont be discouraged by your equipment - I for example only have a 9yr old laptop, 1GB RAM, win XP And one of the cheapest PWS there is This is not important. I do everything in notepad, nothing else and a simple FTP client. You dont need the most fancy powerful PC or Davis

[n9mfk9]

how fast does the size of the sql data base grow my host has a max of  1 GB per database
thanks beau

[Jáchym]

I log mine every 5 mins - which is also the interval I highly recommend to be used.

I measure data since January 1st 2012 and havent missed a single day, so that would be approximately 3 and half yrs almost. Looking at the db size right now it is around 31 MB Which means if you use my recommended interval, you should have something around 106 yrs capacity, I hope that will be enough 

[Jáchym]

And to be even more specific, just made an analysis:

339 991 rows (1 row = 1 set of measurements - Temp, Humidity, Pressure, Rain, Solar, Rainrate, Wind, Gust, Bearing etc.)
data: 16.8 MB
key: 14.7 MB

[Jáchym]

So, just a quick update.

This weekend I managed to create fully automatic scripts that import data into the database - something I was myself worried about how it is going to work and I know how important this is, because many of those who will give my tempalte a try will probably not want to start with a blank database....

Good news is that it seems to be working. My next question is if there are any other sources apart from these (Weather Underground, Weather Display, Cumulus) that you think should be implemented for which I should also create a script. Each script is different, because it has to take into account the specific structure of log files generated by various SW.

The actual import is relatively fast (see screenshot), the only slow one is the WU, which is obvious because you are not getting data from a text file, but rather a webpage and to make it more complicated WU only offers data for one day, not monthly logs, so the script has ti iterate all days in that month.

For each script there is a couple of parameters you have to set, but those are very very simple - for example for WU you have to give your WU ID, for others path to the log file you wish to import. Other than that you only have to set the parameters to import (and there is a tutorial that will guide you through this process) and then set units and optionally limits - in other words if there is some nonsense data you can set limits for each parameter within which the value has to fall to be imported.