Author Topic: archive interval  (Read 9989 times)

0 Members and 1 Guest are viewing this topic.

Offline ncpilot

  • Forecaster
  • *****
  • Posts: 922
    • Monkey Junction Weather
archive interval
« on: August 04, 2006, 05:47:57 PM »
Something I hadn't thought about until recently--but want to make sure I'm understanding it all correctly...

The archive interval, among other things, determines how fast the archive memory will fill up.

Currently I have mine set to 1 minute, which will fill the memory in about 43 hours.

If I had a power outage longer than that time period, without the ability to download, then new data would eventually overwrite old, and I would lose it. Right?

I haven't read anything convincing re: how to set the interval, but in the event of a power outage, it seems it would make sense to reset the interval to something like 5 minutes (good for 8.8 days), while my computer is still running on an UPS?

Is 1 minute intervals useless data? Maybe I should just set it to 5 minutes anyway?
Marc
Wilmington, NC
"Monkey Junction Weather"
Davis VP2 wireless, WeatherLink

Offline ncpilot

  • Forecaster
  • *****
  • Posts: 922
    • Monkey Junction Weather
archive interval
« Reply #1 on: August 04, 2006, 06:36:38 PM »
One thought, I could install weatherlink on my laptop, and then use that to download data if I had a long power outage.

Is there a way to merge data, say a few days of data from the laptop to the full database on the desktop?
Marc
Wilmington, NC
"Monkey Junction Weather"
Davis VP2 wireless, WeatherLink

Offline Anole

  • Forecaster
  • *****
  • Posts: 576
    • http://weather.anolecomputer.com
Re: archive interval
« Reply #2 on: August 04, 2006, 11:34:10 PM »
Quote from: "ncpilot"
Something I hadn't thought about until recently--but want to make sure I'm understanding it all correctly...

The archive interval, among other things, determines how fast the archive memory will fill up.

Currently I have mine set to 1 minute, which will fill the memory in about 43 hours.

If I had a power outage longer than that time period, without the ability to download, then new data would eventually overwrite old, and I would lose it. Right?

I haven't read anything convincing re: how to set the interval, but in the event of a power outage, it seems it would make sense to reset the interval to something like 5 minutes (good for 8.8 days), while my computer is still running on an UPS?

Is 1 minute intervals useless data? Maybe I should just set it to 5 minutes anyway?


You are correct. With the archive interval set to 1 minute you will overwrite the data fairly quickly. 1 minute intervals aren't  "useless" though. They provide a higher resolution of your data. But the trade off is the datalogger only holds so many records. You already seem to have a firm grasp on that concept so I won't go into the details/math.

As with most things, it's a compromise. how "fine" do you want your data vs. how much data do you need to be able to store. I personally feel that the 5-minute archive interval represents the best compromise between data "granularity" and storage capacity.

EDIT: One other thing to consider is that a 1 minute archive interval will make your databases (regardless of which program you use) HUGE! Worth considering if you ever need to edit them.

Offline Mark / Ohio

  • Live from Mars!
  • Forecaster
  • *****
  • Posts: 2401
    • Fairfield County Weather
archive interval
« Reply #3 on: August 04, 2006, 11:51:38 PM »
I have mine set to 15 minutes.  I'm only uploading every 30 min. to CWOP and on the hour for everything else anyway (single phone line, dialup connection).  I believe you still get your absolute highs and lows no matter the archive setting.  Less samples would mainly effect the averages accuracy and visual quality of your graphs.  I started out at 1 hour archive when I bought WeatherLink.  Made the graphs for short time periods look pretty jagged.

I think there are some programs for editing Weatherlink files but I'm not sure if you could merge data with them.
Mark 
2002 Davis VP I Wireless, WeatherLink (Serial), VWS, ImageSalsa, GRLevel3, VirtualVP, VPLive, StartWatch, Windows XP (SP3)


Offline ncpilot

  • Forecaster
  • *****
  • Posts: 922
    • Monkey Junction Weather
archive interval
« Reply #4 on: August 05, 2006, 09:31:09 AM »
Yeah, that's what I tell the women, I have a HUGE database.... :lol:

With disk storage so cheap now per Gig, database size isn't a concern... I did once use my raw data to graph some stuff in Excel, and it definitely has a lot of observation points with a 1 minute interval...

Plus, you can always delete excess data, but it's much more difficult to fill in the gaps--same philosophy in digital photography, start with as much resolution as you can, you can always get rid of it later...

As an aside question, do you ever clear out archive memory? The help file makes it sound like downloads could take longer if the memory is full and you're only downloading the most recent n records for the last time period--because the program has to figure out which are the most recent records... there is no way to automatically clear memory after a download on the Pro model (the option is grayed out on the station setup screen).

I have been thinking of a standby generator--that would solve the power problem... :)

Marc
Marc
Wilmington, NC
"Monkey Junction Weather"
Davis VP2 wireless, WeatherLink

Offline Mark / Ohio

  • Live from Mars!
  • Forecaster
  • *****
  • Posts: 2401
    • Fairfield County Weather
archive interval
« Reply #5 on: August 05, 2006, 05:00:39 PM »
I only clear mine if I get an error trying to download it.
Mark 
2002 Davis VP I Wireless, WeatherLink (Serial), VWS, ImageSalsa, GRLevel3, VirtualVP, VPLive, StartWatch, Windows XP (SP3)


Offline Anole

  • Forecaster
  • *****
  • Posts: 576
    • http://weather.anolecomputer.com
archive interval
« Reply #6 on: August 05, 2006, 05:54:48 PM »
Quote from: "ncpilot"
Yeah, that's what I tell the women, I have a HUGE database.... :lol:

With disk storage so cheap now per Gig, database size isn't a concern... I did once use my raw data to graph some stuff in Excel, and it definitely has a lot of observation points with a 1 minute interval...
Marc


Well keep in mind the actual numbers we're talking about here. At a 1 minute interval you'll have 525600 records for a year. For a 5 minute interval your talking about 105120 records for a year. Unless you are looking at really small time frames (like hours), or your graphs are really, really, really wide, you aren't going to see any difference in the 1 minute vs. 5 minute intervals.

Additionally, if you want to do things with your database in Excel the database size is very relevant. Current versions of Excel will only take up to 65,536 rows of data. At a one minute interval that's only 45.5 days. If all you need is to work with is a month of data that would be ok, but even then you'll have to break the database file up into chunks to work with it. That can be a pain in and of itself regardless of how big the database is and the PITA factor goes up with the size of the database.

With five minute interval you'd be able to work with 227.5 days. Still not a year but better than 45 days.

Offline ncpilot

  • Forecaster
  • *****
  • Posts: 922
    • Monkey Junction Weather
archive interval
« Reply #7 on: August 05, 2006, 06:23:57 PM »
Good points. Another option is to use Access to store the data, and Excel to query the Access database.

In any case, cutting back to 5 minutes seems prudent...
Marc
Wilmington, NC
"Monkey Junction Weather"
Davis VP2 wireless, WeatherLink

Offline Anole

  • Forecaster
  • *****
  • Posts: 576
    • http://weather.anolecomputer.com
archive interval
« Reply #8 on: August 05, 2006, 06:28:47 PM »
An Access db would be a much better way to work with a db file of the sizes we're talking about.

Offline tinplate

  • Forecaster
  • *****
  • Posts: 368
    • http://www.softwx.com/products.html
archive interval
« Reply #9 on: August 05, 2006, 07:23:39 PM »
I like 5 minutes. I used 1 minute for a while, but it seems like overkill since most of the sensors don't change that frequently except for wind. Since the archive data from the console contains the high wind speed over the archive period in addition to the avg wind speed over the archive period, you won't lose the peak gust. Another benefit is that the lower data point density speeds up display of graphs over long time periods. For me, the greatest benefit is that if weatherLink craps out while you're on vacation, you come back to a big hole in your data. Fortunately I switched from 1 to 5 minute archive period a couple months ago. I was out of town for a little over a week recently. I came home and was very displeased to see that WeatherLink had hung several hours after I left town. When I saw that, I quickly killed it and reran it and did a download. The archive was something like 98% full and I didn't lose anything. If I'd been on 1 minute, I'd have lost several days of data which included the record high for the year that was logged while I was away.

Steve

Offline ncpilot

  • Forecaster
  • *****
  • Posts: 922
    • Monkey Junction Weather
archive interval
« Reply #10 on: August 05, 2006, 10:55:46 PM »
8.8 days of data can be stored at 5 minute intervals... (from the handy weatherlink help files...)

I've already switched to 5 minutes...
Marc
Wilmington, NC
"Monkey Junction Weather"
Davis VP2 wireless, WeatherLink

Offline dendrite

  • Senior Member
  • **
  • Posts: 50
archive interval
« Reply #11 on: August 25, 2006, 11:56:47 AM »
I prefer 1-minute, but I run a perl program and use a lot of algorithms and submit my observations to coolwx.com . The 1-minute interval is really needed for the algorithms to function correctly.

If I'm going to be away for a lengthy period I will simply just switch the interval over to 5 minutes while I'm away.

If you are afraid of outages you could always use your laptop during the outage to switch the interval over to 5 minutes until the power comes back on.

 

anything