Author Topic: Weather Stations & Cross loading observation data  (Read 265 times)

0 Members and 1 Guest are viewing this topic.

Offline Mattk

  • Forecaster
  • *****
  • Posts: 1056
Weather Stations & Cross loading observation data
« on: January 19, 2019, 03:36:12 AM »
Bit of a broad ranging query and not specific to any particular make/model etc but what systems out there can import or cross import observation data from what other system's formats?

Offline galfert

  • Forecaster
  • *****
  • Posts: 1218
Re: Weather Stations & Cross loading observation data
« Reply #1 on: January 19, 2019, 06:52:14 AM »
Good question. I'm currently uploading my data on my local network from my Meteobridge to an SQL server (Maria10 db running on Synology NAS). I plan on some day setting up WeeWx and maybe also Weather-Display and importing my SQL data into them. I don't know the specifics of the import process but it should be possible to do so.

So to answer your question you need to look at it from two different angles. It shouldn't really be one question, but rather two questions each with a subquent clarification. 1- Which systems log data that allow you to later retrieve it in some file format (and how much data going back in time is kept)? 2- Which systems allow importing of historical archived data (and if importing is not a direct feature can a compatible database be constructed)?

Between your saved format and subsequent ingesting of that data into another system some manual data manipulation my be required. Like exporting from SQL to CSV file. Or converting CSV file headers to different headers for compatibility. Deleting rows or columns that aren't relevant. Possibly converting units. Possibly altering date field formats....etc.

For example on the part about which systems can you get saved data from... Weather Underground let's you download your data. Just select a date and then ask WU to show the data table. You'll be presented with the option to download. It crudely presents the data in comma delimited form on the browser. You can copy and paste it into a spreadsheet and set some headers and then save as a CSV. That is a lot of manual work for only one day of data. But it is possible and it counts as a way to get data out. Unfortunately if you set the range to show weekly or monthly or yearly data on WU then there is less data points. So depending on the level of detail that you want it can turn out to be a lot of work. The good news is that with WU the storage of historical data does seem to be unlimited going back years.

Lots of other weather websites let you download historical data with less work. WeatherCloud let's you download a CSV for a full month of data but the free account only keeps the last 12 months of data. Less work than WU but limited to 1 year. also only keeps 1 year and they provide a CSV download option also. Several weather computer programs also save to SQL and/or to their own database format. Using your own software is probably the best method to keep historical data. Specifically saving to SQL offers the most flexible method to then be able to export out into whatever custom format you need for some other software or even for an online system that allows uploading.

As for direct importing I haven't fully looked at this yet. But as I mentioned WeeWx and Weather-Display I believe have that functionality. But if a system does not have that direct functionality you can build a database by manually constructing the file format needed and by using naming conventions that it understands then that can be a way to bring data in. For example my new WS-2000 display tablet console can save data to an SD card. There is no direct import function of historical data. But I can look at the format of the data that it uses to save to SD card. It's a simple CSV file. I can see the name that it expects for that file. I can open it in Excel and look at the data and headers. Then I have all the information to build a new file with historical data from my SQL or from WeatherCloud or wherever. And just like that you present the system, without direct import the ability, a way for it to then have historical data that you created manually.

Bottom line is that there is no universal standard for weather data saving like there is for pictures (.JPG). If you want to bring weather data from one place to another you'll have to do some work.

« Last Edit: January 19, 2019, 09:25:53 PM by galfert »
WS-2000 & WS-2902A | ObserverIP | WeatherBridge (Meteobridge)
WU: KFLWINTE111  |  PWSweather: KFLWINTE111
CWOP: FW3708  |  AWEKAS: 14814
Tele-Pole flag pole is here (not installed yet)

Offline Bushman

  • Forecaster
  • *****
  • Posts: 7121
    • Eagle Bay Weather
Re: Weather Stations & Cross loading observation data
« Reply #2 on: January 19, 2019, 11:49:53 AM »