Author Topic: Another Shakeup at CWOP  (Read 10016 times)

0 Members and 1 Guest are viewing this topic.

Offline Mark / Ohio

  • Live from Mars!
  • Forecaster
  • *****
  • Posts: 2500
    • Fairfield County Weather
Another Shakeup at CWOP
« on: February 03, 2008, 11:41:37 PM »
Anyone else been following the list tonight?

I've had my VP live set to the 9 minutes (up from the original 5 minutes when I got DSL a year ago) for a couple of weeks now since I seen Steve's and Ocala's posts.  Today I bumped it up to 11 minutes doing a little experimenting looking at the data at the NWS Missoula site just for my own curiosity.

I don't much understand the technical aspect of it all.  It sounds like they do have a real on going issue and for some reason cannot all come together to solve it.  Also an ongoing problem with un-mentioned, at least for now, software developers and getting news to data contributors.

Sure hope at some point they can get it together.  Maybe the CWOP will have to turn into a business (God forbid an Accuweather or Weatherbug subsidiary) for that to happen.   :sad:

« Last Edit: February 03, 2008, 11:55:17 PM by Mark / Ohio »
Mark 
2002 Davis VP I Wireless, WeatherLink (Serial), GRLevel3, VirtualVP, StartWatch, Weather Display, Windows 10


Offline wmiler

  • Contributor
  • ***
  • Posts: 118
    • http://www.wmiler.org/
Re: Another Shakeup at CWOP
« Reply #1 on: February 04, 2008, 12:23:23 AM »
I do believe a few people are really ticked off.

Unfortunately, I don't think the general CWOP population has particularly noticed since at least 2/3 don't read the list. I personally didn't start reading the list until I moved back in December, and about that time began learning of the problems they were having with system load issues. Since that time, I know a number of developers have stepped up and planned to release the suggested improvements on their next upgrade (WD and wview spring to mind).

On Dave Anderson's point about CWOP participants being a bunch of free loaders, if there was some sort of method to donate to the program like $5(10)/month I know a number of people would prolly be willing to donate.

Offline wmiler

  • Contributor
  • ***
  • Posts: 118
    • http://www.wmiler.org/
Re: Another Shakeup at CWOP
« Reply #2 on: February 04, 2008, 01:25:14 AM »
Well things are getting really screwed up now, I can't even seem to get my data dropped off at all to CWOP.

One suggestion was to switch to Tier2, but that might cause as much, if not more problems with the system. Perhaps it's time for someone else to step up to the plate and provide CWOP with it's own, non-aprs network, UCAR and it's affiliates come to mind.

Offline weatherforyou

  • Forecaster
  • *****
  • Posts: 509
  • My weather is on WeatherForYou.com. Is yours?
    • http://www.weatherforyou.com
Re: Another Shakeup at CWOP
« Reply #3 on: February 04, 2008, 01:37:04 AM »
On Dave Anderson's point about CWOP participants being a bunch of free loaders,
Where did this happen?  I missed that comment!  He thinks the people sending data are free loaders or those sites that use the data without attribution or contribution to CWOP are the free loaders?  If the former, I'll be happy to stop troubling them with my data and just let the data I send through weatherforyou.com get to MADIS that way.  If the latter... I can understand that.  Which is why the weatherforyou.com data provided to MADIS is not to be distributed outside of government or educational uses.  There's very few stations of weatherforyou.com's they use right now, mostly because they're all part of CWOP, too.

... if there was some sort of method to donate to the program like $5(10)/month I know a number of people would prolly be willing to donate.
I'd be glad to pay a monthly fee AS LONG AS other web sites weren't going to be making money from my data without attribution.  Actually we'd probably be able to donate a number of funds, things and services to the program.  After all, we actively promote buying and installing stations to put data on the Internet, so it could be said their success is in some part our fault.  ;)  But any help CWOP would accept from weatherforyou.com may be interpreted as an endorsement by CWOP, or something like that... never quite figured it out.  :???:  Big companies sponsor CPB and it doesn't mean CPB endorses them.  In a way I can see why they would want it to appear as a totally volunteer effort, though.  Guess if help is accepted from AccuWeather or Weatherbug it'll show the program reached some pretty desperate times, which I'd be sorry to hear.  I think it's a great program as it has been and would be glad to help it continue as a paying data contributor if commercial sites wouldn't get most of the benefit.
Joe Torsitano


Offline WeatherHost

  • Forecaster
  • *****
  • Posts: 3189
Re: Another Shakeup at CWOP
« Reply #4 on: February 04, 2008, 08:19:01 AM »
These types of issues are exactly why I don't get involved in those sites.  I send to our own site and to WU.

Banners, tags, widgets, etc. are why I have Signatures turned off.

Offline Anthony

  • Forecaster
  • *****
  • Posts: 1707
    • Anthony's Weather
Re: Another Shakeup at CWOP
« Reply #5 on: February 04, 2008, 09:07:52 AM »
I hardly ever actually go to the CWOP site and check my data. I always rely on the daily error reprting e-mails that I get. Have not seen anything unusual there. Except that I have been getting a few barometric pressure errors. But when I compare with the local Mear. Readings are only of by .01 or .02.



Thanks,
Anthony
WB8YUE

jwyman

  • Guest
Re: Another Shakeup at CWOP
« Reply #6 on: February 04, 2008, 10:08:22 AM »
Guys,
 I must have been asleep for a decade or too much Codiene in the cough syrup but whats going on here? I am missing the gist of the subject matter here about CWOp.. Whats going on? Something we should know about?  Thanks from Mr. Slowness ( I have had the flu for the last 4 days)... :(((((((((((((((


Jim

Offline tinplate

  • Forecaster
  • *****
  • Posts: 368
    • http://www.softwx.com/products.html
Re: Another Shakeup at CWOP
« Reply #7 on: February 04, 2008, 11:33:40 AM »
These types of issues are exactly why I don't get involved in those sites.  I send to our own site and to WU.

I'd like to respond in defense of CWOP. The purpose of CWOP is not to display web pages of weather info like WU. CWOP is a cooperative program that makes weather observations from personal weather stations available to meteorologists and the systems they use. CWOP data adds to the observation density to improve computer forecasting models. NWS local stations often use CWOP data in monitoring developing weather situations. CWOP data is even important in community safety. If there were to be airborne dispersal of hazardous material, there is software that emergency responders use to predict the movement of the plume so that people can be evacuated. The ability of this software to make accurate plume models depends on information such as that provided by CWOP. This data is not just for the benefit of folks looking up the weather on the internet. CWOP data is being used for serious scientific purposes. Once in the MADIS system, it's used by numerous projects. People contributing their data through CWOP are providing a public service, as are those who make it possible for the data to get from the personal weather stations to the MADIS servers. This is a good program, and I hope it does not die.

This is where the CWOP data ends up: http://www-sdd.fsl.noaa.gov/MADIS_Overview/MADIS_Overview.html
Steve
SoftWx
« Last Edit: February 04, 2008, 11:41:23 AM by tinplate »

Offline SLOweather

  • Administrator
  • Forecaster
  • *****
  • Posts: 3434
    • http://www.sloweather.com
Re: Another Shakeup at CWOP
« Reply #8 on: February 04, 2008, 11:59:36 AM »
Here's the background (long) as posted to another forum to which I belong.

---


From: Dave Anderson KG4YZY
>   To: 'Discussion of weather data quality issues'
>   Sent: Sunday, February 03, 2008 8:36 PM
>   Subject: [wxqc] Unexpected loss of 1/2 the servers that support CWOP.
>
>
>   
>
>   Hello all,
>
>   
>
>   I am here to regretfully announce that the third.aprs.net and
> fourth.aprs.net servers that I personally own and operate will no
> longer be supporting the CWOP program.
>
>   
>
>   Allow me to give a brief history on events, so you understand my decision.
>
>   
>
>   I am a ham radio operator.  The APRS-IS network was built by hams, for hams.   
> CWOP, non ham users, are guests on our network to get to MADIS.   CWOP users
> account for 1/8 of the total station count on APRS, the other 7/8 are
> non weather ham radio users.  The traffic is disproportionate though,
> as CWOP users take up a full < of the bandwidth passing thru the servers.
>
>   
>
>   Back January of last year, the previous group of server sysops that
> had ran the servers for CWOP since day one abruptly pulled support for
> the CWOP program one Friday.  No excuse was given beyond that it was
> in the best interest of both parties.  This was a devastating blow to CWOP.
>
>   
>
>   The core group of server sysops were contacted about this, and we
> spung into action to help get CWOP taken care of.  It required extensive work to get this
> to happen, as we had only three servers at the time.    In less than 48 hours we
> managed to get a 4th server online, and had started moving users over.
>
>   
>
>   Shortly after this, I observed that the method that the stations
> were using to submit the data was horribly wrong for the APRS-IS
> network (does not follow the spec), knowing that as the load
> increased, we would have problems.  I started writing a white paper,
> which I have sent to several of you on request, that outlined
> everything a developer needed to know to write a program to use the APRS-IS network without causing undue network load.
>
>   
>
>   During the development of this white paper there were 8 if I recall
> folks that were involved with the "fine tuning" of it.  At the time,
> we wanted to set the host name for CWOP users to use as cwop.aprs.net,
> so later we could move you guys to your own dedicated network of
> servers (thinking ahead).  There were also a few other items that we wanted addressed, such as the polling time, and
> polling interval.     Dave Helms, one of the CWOP administrators refused to
> accept some of the items in the draft.   After several attempts to talk reason
> into him, I just said the heck with it, and let it sit in my temp
> directory, never finished.  No one else picked up the ropes to keep
> going, either, and everyone had the last version of the draft!
>
>   
>
>   Here's the problems. The core sysops have been begging Dave to get
> the developers to fix their software.  Dave, fearing that developers
> would pull out of the CWOP program with the changes the core was
> wanting to be made, has had very little done about this.  When most of
> you have problems with data loss, or difficulty in connecting to a
> server, it's because the policies that the core put in place were
> never implemented by most of the software developers because Dave never passed it on to them.  We can only operate a network so long as the
> software -you- use is set to work with it correctly.    So a year later, after
> suggestions have been made, most of you are still using software that
> is broken, or have incorrect settings in place to work 100% correctly with the servers.
> And we've been asking over, and over, and over for these changes to be
> implemented, as even as far back as October we started noticing a
> detrimental impact on the network, but nothing has been done.
>
>   
>
>   Jump to today, here's the problems that still exist that are causing
> server sysops problems and are causing you problems due to the fact
> that most of the CWOP programs don't follow the rules:
>
>   
>
>   The lion share of you have software that uses your PC clock for
> polling time, meaning our servers have to accept thousands of stations
> every 5 minutes then sit relatively idle for 4 more minutes.  This
> with the addition of new Christmas gift stations this year has been
> causing countless crashes, data loss, and other issues that  the
> server sysops have been working on.  Each server can only accept 50
> backlogged connections at a shot, and frequently port 14580 way exceeds that.  So it's caused instabilities in the network, as networking queue
> times go very high when this happens affecting HAMS as well as CWOP.   We've
> asked this to be fixed, and heard silence.
>
>   
>
>   Polling interval is another issue.  MADIS takes the data from the
> APRS-IS network via Findu only every 15 minutes, but we still have
> stations that were sending data in every minute.  Keep in mind, you -don't stay- connected to the
> server, you connect, drop off, and disconnect.    To Russ Chadwick's credit
> action was finally taken just here recently about these stations, and
> we've banned them at the server level, but that still doesn't prevent
> them from going to connect, and tying up a server slot every minute.
> 5 minutes is the standard that amateur radio operators have used on
> APRS for probably 10 years now.  Why that wasn't just hard coded in
> programs, I don't know, but again, this is another issue we've been asking over and over to get fixed, and it yet is.
>
>   
>
>   Round Robin DNS still doesn't work for the most of you.  Over = of
> the CWOP users use third alone.  Many of the programs simply only look
> up the IP address of the rotate.aprs.net when you first load the
> program or connect the first time, and then it -never- goes back to
> ask if the IP has changed, or check if there is another one for load
> balancing.  Again, asked over and over for this to be addresses with the developers, still hasn't been.
>
>   
>
>   I am by no means saying that every piece of CWOP software is broken!!!
> Numerous apps -do it right- as the developers of those programs have
> been in this forum, seen some of our technical posts about problems
> and corrected their code.  But the amount of developers who seem to
> care about doing it right is very few, and the largest CWOP
> applications that the bulk of the users use, simply do not follow the rules.
>
>   
>
>   So who suffers?  We all do.  As the server sysops, we have to sit,
> being sucker punched every 5 minutes right now.  You do as not all of
> your weather data makes it thru.
>
>   
>
>   APRS-IS is the internet side of a VHF radio system on 144.390mhz
> that does packet radio.  Weather is only a small fraction of APRS's
> overall functionality, so as system operators we have to be sensitive
> to this, not to allow CWOP to cause harm to the amateur radio
> operators that rely on this for emergency communications and safety of life traffic.
>
>   
>
>   The servers are ran by a 100% volunteer group of system operators
> that in the case of the core servers, are all ran in datacenters.
> Fourth alone is a 8 core Xeon box running VMWare ESX server with
> Windows 2003 running the fourth virtual machine on that box.  I spent
> $4500.00 for that server to -SUPPORT- cwop.  I also donate about 6 meg
> of tier1 quality backbone bandwidth every day to this program.  Unlike
> Findu, where Steve gets support from his users of that website, the
> server sysops, who without there would be no network to use, we get
> zero support from the user community.  That' after thousands of dollars in hardware and hundreds of hours of time.
>
>   
>
>   So how was I repaid for my generosity?Huh?
>
>   
>
>   7         Requests to make changes to the client software went unanswered,
> allowing things to spiral to where they are now.
>
>   7        Every time anything happens to the network, the server sysops are
> automatically crucified for doing something wrong.
>
>   7        I can't think of the last time anyone stopped to tell me how much
> they appreciate what I do.
>
>   
>
>   So it all boils down to this morning, when I posed the message at
> 10:24 am to Seth about the standing of the software problems.
>
>   
>
>   Shortly afterward, I go this from Dave Helms:
>
>   
>
>   
>
>   Dave,
>
>   
>
>   I, too, have absolutely no interest in revisiting the events of
> January-March
> 2007 (publicly or otherwise).  I am willing to support the notion of
> segmenting CW traffic to improve stability of APRS-IS, to include developing a plan for
> implementing the cwop.aprs.net round robin server name.   In last year's crisis
> mode, I agreed to contact virtually all CW and Ham weather station
> operators to facilitate the switch from Tier2 servers to Core servers
> included on the rotate.aprs.net RR server domain.  In the future, I
> will rely on encouraging individuals to read the CWOP News Page to
> advertise the RR server name change, and through routine client
> application upgrades rather than making individual email contacts.
>
>   
>
>   As we move forward, I request one thing... that we not make the "CW'
> (non-Ham) volunteers second class users.  Their contributions are equally valuable to NOAA
> as validated Ham Radio users who send weather reports to APRS-IS and NOAA.   If
> weather traffic must be (temporarily) throttled back to preserve
> APRS-IS stability, then all weather contributions should be equally
> reduced, Ham and non-Hams alike.
>
>   
>
>   Going forward, we need to find a way to add capacity to the APRS-IS
> to take the load of the expected growing number of users.  Hence, we
> need to develop an architecture that will allow exploitation of both
> Ham Radio APRS-IS server resources and non-Ham servers which may be
> available. These non-Ham servers and bandwidth may be donated from
> government, universities, and private sector groups.
>
>   
>
>   Regards,
>
>   
>
>   Dave
>
>   
>
>   
>
>   
>
>   
>
>   Finally after a year, Dave concedes to allowing us to start using
> CWOP.APRS.NET.  Well the problem is that it's a day late, and a dollar short.
> He further more says it'll only be a news post on a website, and for
> when folks update software.  Well if the last change to rotate is any
> indication of how well that will go, there still are 800 of you using the old servers to this DAY.
> So if he's not going to help promote it, we'll have a network that is
> in shambles here in the next 90 days due to load.  I just don't think
> Dave understands the gravity of the problem.
>
>   
>
>   Then the straw that broke the camel's back was in the second paragraph.
> Insisting that ham stations also be reduced to that of non hams.  Well first
> off, ham stations are generally sharing a very limited amount of bandwidth on
> 144.390mhz (1200bps).  From peer pressure alone, no ham will beacon weather more
> than every 5 minutes without hearing about it from other local hams.  We have a
> system in place that works well, using nothing but peer pressure.
>
>   
>
>   The fact that he's insisting hams back off, on their own network, that was
> built by  hams, really offended me, and so far most of the hams' I've shared
> this with seem to agree.  Dave and the NWS pay NOTHING for what we do on the
> network.  How dare he tell us what to do on our own network.  CWOP are GUESTS on
> our network.  Dave has in the past shown callous towards the hams, never this
> blatant, but I've seen it in other messages.  This disgusts me.  This would be
> no different than me coming into your house as a guest and telling you can only
> spend 5 hours a day watching TV.
>
>   
>
>   So I write Dave back a -very- long message, and at this point I'm very upset
> about this.  I lay blame for the problems we have right now where it needs to be
> placed, squarely on his shoulders.  He was the reason the white paper was given
> up on, as if it wasn't Dave's way, it was no good.  Yeah, I'll admit my message
> to him was long, and rather blunt.  I was direct with him, and factually telling
> him it's come to a point where change has to occur, or the network will change
> around him.
>
>   
>
>   Well, here was his response to that:
>
>   
>
>   
>
>   
>
>   
>
>   
>
>   Dave,
>
>   
>
>   In the time your spent wiring this email, you would could have instead been
> making progress on the a white paper your said you would complete last year.   
> Don't blame the client developers for something you knew that was needed over a
> year ago.
>
>   
>
>   Dave Helms
>
>   
>
>   
>
>   
>
>   
>
>   Accusing me of not finishing a document that he opposed?!!?!?!?!?!!  Excuse
> me?  Draft 3 was what would have gotten published had he not objected to the
> hostname.  The rest -is- done right now.
>
>   
>
>   Well if the NWS employee that is supposed to be our liaison to the developers
> is taking this attitude towards me, with the thousands I have spent to support
> CWOP, I hereby pull my official support from the program.
>
>   
>
>   I realize this will be a heavy hit for CWOP, as I take a full one half of the
> server capacity with me.   I am greatly sorry for that.  The last people I
> wanted to hurt were the ones I was out to help, but with Dave Helms at the
> "Helm", it's over for me.  I cannot continue to be badgered, beaten, ignored,
> then finally blamed for it to begin with over a volunteer operation.
>
>   
>
>   If any of you would like to talk about this more, I encourage you to send me a
> private e-mail at dave@aprsfl.net.  I'm sure this message will get me banned on
> this forum, so I'll not be checking back.
>
>   
>
>   I'm glad I've been able to help up till now, and I really am truly sorry for
> going out this way, but I just can't go on with this type of pervasive
> attitude.. Let alone one that is so callous towards the folks that this network
> was built for, ham radio operators.
>
>   
>
>   Regards,
>
>   Dave Anderson
>
>   KG4YZY

Offline weatherforyou

  • Forecaster
  • *****
  • Posts: 509
  • My weather is on WeatherForYou.com. Is yours?
    • http://www.weatherforyou.com
Re: Another Shakeup at CWOP
« Reply #9 on: February 04, 2008, 12:47:34 PM »
Wow!   :shock:

I e-mailed Russ Chadwick offering assistance from weatherforyou.com, whether it be servers or whatever.

I'm wondering more about how their system is designed.  They mentioned round-robin DNS, which I found a long time ago doesn't work well.  Although we still use round-robin to a small extent, mostly for redundancy, there are much more sophisticated load balancing systems that are free for Linux servers.  HAProxy is one we've used for quite some time.
Joe Torsitano


Offline wmiler

  • Contributor
  • ***
  • Posts: 118
    • http://www.wmiler.org/
Re: Another Shakeup at CWOP
« Reply #10 on: February 04, 2008, 07:10:39 PM »
Wow!   :shock:

I e-mailed Russ Chadwick offering assistance from weatherforyou.com, whether it be servers or whatever.

I'm wondering more about how their system is designed.  They mentioned round-robin DNS, which I found a long time ago doesn't work well.  Although we still use round-robin to a small extent, mostly for redundancy, there are much more sophisticated load balancing systems that are free for Linux servers.  HAProxy is one we've used for quite some time.


The problem as I understand it, isn't that they need more servers, but rather that the current data providers (ie CWOP sites) are:
a) slamming the core servers every 5 to 10 minutes (1:00, 1:05, 1:10, etc) and not spreading the load across     
    all the available time slots.
b) the way aprs is designed, every time I send a report in, aprs turns the report around and sends it back out
    to 300-1300 ham operators.

In the case of (b), this is a terrible way for CWOP to run a network, but it was the nature of the beast when CWOP was founded. Simple example, 3500 CW stations nail one of four servers at 8:05 am, aprs turns around, and pushes each one of those reports back out to 300-1300 hams within 4.5 minutes before the server is hammered again at 8:10am. APRS2 claims they may have as many as 1300 operators seeking data world-wide at any point in time.

Offline wuhu_software

  • Forecaster
  • *****
  • Posts: 688
    • WUHU Software Yahoo Group
Re: Another Shakeup at CWOP
« Reply #11 on: February 04, 2008, 07:54:48 PM »
Wow!   :shock:

I e-mailed Russ Chadwick offering assistance from weatherforyou.com, whether it be servers or whatever.

I'm wondering more about how their system is designed.  They mentioned round-robin DNS, which I found a long time ago doesn't work well.  Although we still use round-robin to a small extent, mostly for redundancy, there are much more sophisticated load balancing systems that are free for Linux servers.  HAProxy is one we've used for quite some time.

The problem as I understand it, isn't that they need more servers, but rather that the current data providers (ie CWOP sites) are:
a) slamming the core servers every 5 to 10 minutes (1:00, 1:05, 1:10, etc) and not spreading the load across     
    all the available time slots.
b) the way aprs is designed, every time I send a report in, aprs turns the report around and sends it back out
    to 300-1300 ham operators.

In the case of (b), this is a terrible way for CWOP to run a network, but it was the nature of the beast when CWOP was founded. Simple example, 3500 CW stations nail one of four servers at 8:05 am, aprs turns around, and pushes each one of those reports back out to 300-1300 hams within 4.5 minutes before the server is hammered again at 8:10am. APRS2 claims they may have as many as 1300 operators seeking data world-wide at any point in time.

I am not sure, but I would guess they not only push the data to their connected clients, but they also push the data to all of the other servers, which in turn push the data to all of their directly connected clients. So basically, if you post to any server, it ends up being pushed to everyone that is not connected to a "filtered" port (no unsolicited traffic).



Offline edpnjax

  • Senior Member
  • **
  • Posts: 94
    • RiverCityWeather.Net
Re: Another Shakeup at CWOP
« Reply #12 on: February 04, 2008, 08:28:32 PM »
These types of issues are exactly why I don't get involved in those sites.  I send to our own site and to WU.

I'd like to respond in defense of CWOP. The purpose of CWOP is not to display web pages of weather info like WU. CWOP is a cooperative program that makes weather observations from personal weather stations available to meteorologists and the systems they use. CWOP data adds to the observation density to improve computer forecasting models. NWS local stations often use CWOP data in monitoring developing weather situations. CWOP data is even important in community safety. If there were to be airborne dispersal of hazardous material, there is software that emergency responders use to predict the movement of the plume so that people can be evacuated. The ability of this software to make accurate plume models depends on information such as that provided by CWOP. This data is not just for the benefit of folks looking up the weather on the internet. CWOP data is being used for serious scientific purposes. Once in the MADIS system, it's used by numerous projects. People contributing their data through CWOP are providing a public service, as are those who make it possible for the data to get from the personal weather stations to the MADIS servers. This is a good program, and I hope it does not die.

This is where the CWOP data ends up: http://www-sdd.fsl.noaa.gov/MADIS_Overview/MADIS_Overview.html
Steve
SoftWx

Does VPLive yet support any of the changes they are (have been) asking for? From what I can see, it does not support any way to to set an upload time slot. Does it use the round-robin DNS properly, ie not only lookup at startup, but lookup at every send time so it "load balances"?
Emmett


Weather for Jacksonville / St Johns FL
Davis VP2 Plus w/24hr FARS | Live Weather | Boltek / NexStorm | GRLevel3

Offline wmiler

  • Contributor
  • ***
  • Posts: 118
    • http://www.wmiler.org/
Re: Another Shakeup at CWOP
« Reply #13 on: February 04, 2008, 08:50:55 PM »
The problem as I understand it, isn't that they need more servers, but rather that the current data providers (ie CWOP sites) are:
a) slamming the core servers every 5 to 10 minutes (1:00, 1:05, 1:10, etc) and not spreading the load across     
    all the available time slots.
b) the way aprs is designed, every time I send a report in, aprs turns the report around and sends it back out
    to 300-1300 ham operators.

In the case of (b), this is a terrible way for CWOP to run a network, but it was the nature of the beast when CWOP was founded. Simple example, 3500 CW stations nail one of four servers at 8:05 am, aprs turns around, and pushes each one of those reports back out to 300-1300 hams within 4.5 minutes before the server is hammered again at 8:10am. APRS2 claims they may have as many as 1300 operators seeking data world-wide at any point in time.

I am not sure, but I would guess they not only push the data to their connected clients, but they also push the data to all of the other servers, which in turn push the data to all of their directly connected clients. So basically, if you post to any server, it ends up being pushed to everyone that is not connected to a "filtered" port (no unsolicited traffic).

I believe that is correct, from what I have read. So in that case the network, acting as a relay, spews even more traffic out in a snowball fashion. As I recall, Core has 4 servers, which feed Tier2 which has 37 servers.

Offline tinplate

  • Forecaster
  • *****
  • Posts: 368
    • http://www.softwx.com/products.html
Re: Another Shakeup at CWOP
« Reply #14 on: February 04, 2008, 10:01:14 PM »
Quote
Does VPLive yet support any of the changes they are (have been) asking for? From what I can see, it does not support any way to to set an upload time slot. Does it use the round-robin DNS properly, ie not only lookup at startup, but lookup at every send time so it "load balances"?

VPLive doesn't yet support setting specific time slots. However, you can achieve most of what they want from the recommended changes by setting the APRS upload frequency in VPLive to either 9 or 11 minutes. This will distribute the timing so it doesn't always fall on 5 or 10 minute boundaries, as well as put the frequency near 10 minutes.

VPLive already uses the rotate.aprs.net dns entry as previously recommended, by resolving a new IP address each time it sends to APRS. VPLive also has a fallback algorithm that tries a list of specific servers if the server returned by rotate doesn't work. If you look at your raw APRS traffic on findu, and you're using VPLive you'll probably see that VPLive occasionally has to use tier 2 servers to get the data through since the first and second core servers are now overloaded.

I'm working on changes I hope to have out soon that respond to these current issues. One of the things I'm doing, since the situation is in flux, is to make it so VPLive occasionally downloads a "best practices" file from the SoftWx website. This will allow VPLive to automatically change with landscape without making users upgrade to a new version. This file will contain the name(s) of the rotation dns entries, as well as specific servers for fallback.

Steve

Offline Mark / Ohio

  • Live from Mars!
  • Forecaster
  • *****
  • Posts: 2500
    • Fairfield County Weather
Re: Another Shakeup at CWOP
« Reply #15 on: February 04, 2008, 11:33:11 PM »
....VPLive already uses the rotate.aprs.net dns entry as previously recommended, by resolving a new IP address each time it sends to APRS. VPLive also has a fallback algorithm that tries a list of specific servers if the server returned by rotate doesn't work. If you look at your raw APRS traffic on findu, and you're using VPLive you'll probably see that VPLive occasionally has to use tier 2 servers to get the data through since the first and second core servers are now overloaded.

Looks like that it is working exactly as planned.  I don't think it has missed an upload yet on my end.  Here's a snip of my uploads:

CW3403>APRS,TCPXX*,qAX,CORE-2:@040428z3945.22N/08234.10W_095/003g006t032r000P000h90b10172VL1181
CW3403>APRS,TCPXX*,qAX,CORE-2:@040439z3945.22N/08234.10W_093/000g003t032r000P000h89b10168VL1181
CW3403>APRS,TCPXX*,qAX,T2APRSWST:@040450z3945.22N/08234.10W_067/000g002t032r000P000h88b10175VL1181
CW3403>APRS,TCPXX*,qAX,CORE-2:@040459z3945.22N/08234.10W_068/000g003t032r000P000h86b10174VL1181
CW3403>APRS,TCPXX*,qAX,CORE-2:@040511z3945.22N/08234.10W_066/000g003t033r000P000h85b10175VL1181
CW3403>APRS,TCPXX*,qAX,CORE-2:@040522z3945.22N/08234.10W_033/001g002t033r000P000h86b10170VL1181
CW3403>APRS,TCPXX*,qAX,CORE-2:@040533z3945.22N/08234.10W_063/001g003t032r000P000h85b10175VL1181
CW3403>APRS,TCPXX*,qAX,CORE-2:@040544z3945.22N/08234.10W_066/000g000t032r000P000h87b10175VL1181
CW3403>APRS,TCPXX*,qAX,FIRST:@040555z3945.22N/08234.10W_034/001g003t033r000P000h86b10172VL1181
CW3403>APRS,TCPXX*,qAX,CORE-2:@040606z3945.22N/08234.10W_017/002g004t032r000P000h87b10172VL1181
CW3403>APRS,TCPXX*,qAX,CORE-2:@040617z3945.22N/08234.10W_045/000g005t032r000P000h88b10168VL1181

....One of the things I'm doing, since the situation is in flux, is to make it so VPLive occasionally downloads a "best practices" file from the SoftWx website. This will allow VPLive to automatically change with landscape without making users upgrade to a new version. This file will contain the name(s) of the rotation dns entries, as well as specific servers for fallback.

That sounds like a good idea.  The way I read their setup at the moment, it's like all of our individual data is being sent back out to hundreds if not thousands of weather offices due to it being tied together with the HAM network.  How many of those places that it is going really needs or wants our individual data?  Seems to me like at some point they will be forced to seperate the two for both to survive being operated on a non profit / volunteer status.
« Last Edit: February 04, 2008, 11:38:24 PM by Mark / Ohio »
Mark 
2002 Davis VP I Wireless, WeatherLink (Serial), GRLevel3, VirtualVP, StartWatch, Weather Display, Windows 10


Offline edpnjax

  • Senior Member
  • **
  • Posts: 94
    • RiverCityWeather.Net
Re: Another Shakeup at CWOP
« Reply #16 on: February 06, 2008, 08:13:13 PM »
Quote
I'm working on changes I hope to have out soon that respond to these current issues. One of the things I'm doing, since the situation is in flux, is to make it so VPLive occasionally downloads a "best practices" file from the SoftWx website. This will allow VPLive to automatically change with landscape without making users upgrade to a new version. This file will contain the name(s) of the rotation dns entries, as well as specific servers for fallback.

Thanks for the update Steve. I'll go along with Mark, your idea for the best practices sounds like an excellent idea!
Emmett


Weather for Jacksonville / St Johns FL
Davis VP2 Plus w/24hr FARS | Live Weather | Boltek / NexStorm | GRLevel3

Offline Mark / Ohio

  • Live from Mars!
  • Forecaster
  • *****
  • Posts: 2500
    • Fairfield County Weather
Re: Another Shakeup at CWOP
« Reply #17 on: February 06, 2008, 08:30:56 PM »
I feel sorry for Russ Chadwick.  I'd offer to give him a bigger ruler to slap the wrists of a few kids but keeping upper level folk separated should not have to be his job anyway.   :roll:

Stayed tuned for tomorrows edition of "As the Weather Observer Turns" as our story continues...   :-({|=


 ](*,)
Mark 
2002 Davis VP I Wireless, WeatherLink (Serial), GRLevel3, VirtualVP, StartWatch, Weather Display, Windows 10


Offline Cienega32

  • Forecaster
  • *****
  • Posts: 2571
    • East Mesa Weather
Re: Another Shakeup at CWOP
« Reply #18 on: February 06, 2008, 11:56:52 PM »
WL 5.7.1 won't send hardly at all with the 'rotate' but after I changed it to the 'cwop' that was talked about in that white paper, the problem has minimized to only now & then.

Pat ~ Davis VP2 6153-Weatherlink-Weather Display-StartWatch-VirtualVP-Win7 Pro-64bit
www.LasCruces-Weather.com   www.EastMesaWeather.com

Offline jgillett

  • Forecaster
  • *****
  • Posts: 1155
  • unnamed weather station, Boltek, Win7 Pro, ToA
    • TiggrWeather Phoenix
Re: Another Shakeup at CWOP
« Reply #19 on: March 25, 2008, 08:52:28 AM »
I was off CWOP for a while, but recently began sending data again when I switched over to VWS software. Could somebody kindly point me to the list you all watch so I can get more info on what's going on?

Thanks.
John
W7JKG
BUY FROM AMAZON  HERE... Thank You! :-)

Offline jmcmurry

  • Jim McMurry
  • Forecaster
  • *****
  • Posts: 475
  • Davis Vantage Pro 2 Plus Wireless.
    • Juneau County Weather
Re: Another Shakeup at CWOP
« Reply #20 on: March 25, 2008, 09:12:10 AM »

Forum Search and Google Can be Your Best Friends

Offline jgillett

  • Forecaster
  • *****
  • Posts: 1155
  • unnamed weather station, Boltek, Win7 Pro, ToA
    • TiggrWeather Phoenix
Re: Another Shakeup at CWOP
« Reply #21 on: March 25, 2008, 10:19:36 AM »
Thanks very much, Jim.
John
W7JKG
BUY FROM AMAZON  HERE... Thank You! :-)