Here's the background (long) as posted to another forum to which I belong.
---
From: Dave Anderson KG4YZY
> To: 'Discussion of weather data quality issues'
> Sent: Sunday, February 03, 2008 8:36 PM
> Subject: [wxqc] Unexpected loss of 1/2 the servers that support CWOP.
>
>
>
>
> Hello all,
>
>
>
> I am here to regretfully announce that the third.aprs.net and
> fourth.aprs.net servers that I personally own and operate will no
> longer be supporting the CWOP program.
>
>
>
> Allow me to give a brief history on events, so you understand my decision.
>
>
>
> I am a ham radio operator. The APRS-IS network was built by hams, for hams.
> CWOP, non ham users, are guests on our network to get to MADIS. CWOP users
> account for 1/8 of the total station count on APRS, the other 7/8 are
> non weather ham radio users. The traffic is disproportionate though,
> as CWOP users take up a full < of the bandwidth passing thru the servers.
>
>
>
> Back January of last year, the previous group of server sysops that
> had ran the servers for CWOP since day one abruptly pulled support for
> the CWOP program one Friday. No excuse was given beyond that it was
> in the best interest of both parties. This was a devastating blow to CWOP.
>
>
>
> The core group of server sysops were contacted about this, and we
> spung into action to help get CWOP taken care of. It required extensive work to get this
> to happen, as we had only three servers at the time. In less than 48 hours we
> managed to get a 4th server online, and had started moving users over.
>
>
>
> Shortly after this, I observed that the method that the stations
> were using to submit the data was horribly wrong for the APRS-IS
> network (does not follow the spec), knowing that as the load
> increased, we would have problems. I started writing a white paper,
> which I have sent to several of you on request, that outlined
> everything a developer needed to know to write a program to use the APRS-IS network without causing undue network load.
>
>
>
> During the development of this white paper there were 8 if I recall
> folks that were involved with the "fine tuning" of it. At the time,
> we wanted to set the host name for CWOP users to use as cwop.aprs.net,
> so later we could move you guys to your own dedicated network of
> servers (thinking ahead). There were also a few other items that we wanted addressed, such as the polling time, and
> polling interval. Dave Helms, one of the CWOP administrators refused to
> accept some of the items in the draft. After several attempts to talk reason
> into him, I just said the heck with it, and let it sit in my temp
> directory, never finished. No one else picked up the ropes to keep
> going, either, and everyone had the last version of the draft!
>
>
>
> Here's the problems. The core sysops have been begging Dave to get
> the developers to fix their software. Dave, fearing that developers
> would pull out of the CWOP program with the changes the core was
> wanting to be made, has had very little done about this. When most of
> you have problems with data loss, or difficulty in connecting to a
> server, it's because the policies that the core put in place were
> never implemented by most of the software developers because Dave never passed it on to them. We can only operate a network so long as the
> software -you- use is set to work with it correctly. So a year later, after
> suggestions have been made, most of you are still using software that
> is broken, or have incorrect settings in place to work 100% correctly with the servers.
> And we've been asking over, and over, and over for these changes to be
> implemented, as even as far back as October we started noticing a
> detrimental impact on the network, but nothing has been done.
>
>
>
> Jump to today, here's the problems that still exist that are causing
> server sysops problems and are causing you problems due to the fact
> that most of the CWOP programs don't follow the rules:
>
>
>
> The lion share of you have software that uses your PC clock for
> polling time, meaning our servers have to accept thousands of stations
> every 5 minutes then sit relatively idle for 4 more minutes. This
> with the addition of new Christmas gift stations this year has been
> causing countless crashes, data loss, and other issues that the
> server sysops have been working on. Each server can only accept 50
> backlogged connections at a shot, and frequently port 14580 way exceeds that. So it's caused instabilities in the network, as networking queue
> times go very high when this happens affecting HAMS as well as CWOP. We've
> asked this to be fixed, and heard silence.
>
>
>
> Polling interval is another issue. MADIS takes the data from the
> APRS-IS network via Findu only every 15 minutes, but we still have
> stations that were sending data in every minute. Keep in mind, you -don't stay- connected to the
> server, you connect, drop off, and disconnect. To Russ Chadwick's credit
> action was finally taken just here recently about these stations, and
> we've banned them at the server level, but that still doesn't prevent
> them from going to connect, and tying up a server slot every minute.
> 5 minutes is the standard that amateur radio operators have used on
> APRS for probably 10 years now. Why that wasn't just hard coded in
> programs, I don't know, but again, this is another issue we've been asking over and over to get fixed, and it yet is.
>
>
>
> Round Robin DNS still doesn't work for the most of you. Over = of
> the CWOP users use third alone. Many of the programs simply only look
> up the IP address of the rotate.aprs.net when you first load the
> program or connect the first time, and then it -never- goes back to
> ask if the IP has changed, or check if there is another one for load
> balancing. Again, asked over and over for this to be addresses with the developers, still hasn't been.
>
>
>
> I am by no means saying that every piece of CWOP software is broken!!!
> Numerous apps -do it right- as the developers of those programs have
> been in this forum, seen some of our technical posts about problems
> and corrected their code. But the amount of developers who seem to
> care about doing it right is very few, and the largest CWOP
> applications that the bulk of the users use, simply do not follow the rules.
>
>
>
> So who suffers? We all do. As the server sysops, we have to sit,
> being sucker punched every 5 minutes right now. You do as not all of
> your weather data makes it thru.
>
>
>
> APRS-IS is the internet side of a VHF radio system on 144.390mhz
> that does packet radio. Weather is only a small fraction of APRS's
> overall functionality, so as system operators we have to be sensitive
> to this, not to allow CWOP to cause harm to the amateur radio
> operators that rely on this for emergency communications and safety of life traffic.
>
>
>
> The servers are ran by a 100% volunteer group of system operators
> that in the case of the core servers, are all ran in datacenters.
> Fourth alone is a 8 core Xeon box running VMWare ESX server with
> Windows 2003 running the fourth virtual machine on that box. I spent
> $4500.00 for that server to -SUPPORT- cwop. I also donate about 6 meg
> of tier1 quality backbone bandwidth every day to this program. Unlike
> Findu, where Steve gets support from his users of that website, the
> server sysops, who without there would be no network to use, we get
> zero support from the user community. That' after thousands of dollars in hardware and hundreds of hours of time.
>
>
>
> So how was I repaid for my generosity?Huh?
>
>
>
> 7 Requests to make changes to the client software went unanswered,
> allowing things to spiral to where they are now.
>
> 7 Every time anything happens to the network, the server sysops are
> automatically crucified for doing something wrong.
>
> 7 I can't think of the last time anyone stopped to tell me how much
> they appreciate what I do.
>
>
>
> So it all boils down to this morning, when I posed the message at
> 10:24 am to Seth about the standing of the software problems.
>
>
>
> Shortly afterward, I go this from Dave Helms:
>
>
>
>
>
> Dave,
>
>
>
> I, too, have absolutely no interest in revisiting the events of
> January-March
> 2007 (publicly or otherwise). I am willing to support the notion of
> segmenting CW traffic to improve stability of APRS-IS, to include developing a plan for
> implementing the cwop.aprs.net round robin server name. In last year's crisis
> mode, I agreed to contact virtually all CW and Ham weather station
> operators to facilitate the switch from Tier2 servers to Core servers
> included on the rotate.aprs.net RR server domain. In the future, I
> will rely on encouraging individuals to read the CWOP News Page to
> advertise the RR server name change, and through routine client
> application upgrades rather than making individual email contacts.
>
>
>
> As we move forward, I request one thing... that we not make the "CW'
> (non-Ham) volunteers second class users. Their contributions are equally valuable to NOAA
> as validated Ham Radio users who send weather reports to APRS-IS and NOAA. If
> weather traffic must be (temporarily) throttled back to preserve
> APRS-IS stability, then all weather contributions should be equally
> reduced, Ham and non-Hams alike.
>
>
>
> Going forward, we need to find a way to add capacity to the APRS-IS
> to take the load of the expected growing number of users. Hence, we
> need to develop an architecture that will allow exploitation of both
> Ham Radio APRS-IS server resources and non-Ham servers which may be
> available. These non-Ham servers and bandwidth may be donated from
> government, universities, and private sector groups.
>
>
>
> Regards,
>
>
>
> Dave
>
>
>
>
>
>
>
>
>
> Finally after a year, Dave concedes to allowing us to start using
> CWOP.APRS.NET. Well the problem is that it's a day late, and a dollar short.
> He further more says it'll only be a news post on a website, and for
> when folks update software. Well if the last change to rotate is any
> indication of how well that will go, there still are 800 of you using the old servers to this DAY.
> So if he's not going to help promote it, we'll have a network that is
> in shambles here in the next 90 days due to load. I just don't think
> Dave understands the gravity of the problem.
>
>
>
> Then the straw that broke the camel's back was in the second paragraph.
> Insisting that ham stations also be reduced to that of non hams. Well first
> off, ham stations are generally sharing a very limited amount of bandwidth on
> 144.390mhz (1200bps). From peer pressure alone, no ham will beacon weather more
> than every 5 minutes without hearing about it from other local hams. We have a
> system in place that works well, using nothing but peer pressure.
>
>
>
> The fact that he's insisting hams back off, on their own network, that was
> built by hams, really offended me, and so far most of the hams' I've shared
> this with seem to agree. Dave and the NWS pay NOTHING for what we do on the
> network. How dare he tell us what to do on our own network. CWOP are GUESTS on
> our network. Dave has in the past shown callous towards the hams, never this
> blatant, but I've seen it in other messages. This disgusts me. This would be
> no different than me coming into your house as a guest and telling you can only
> spend 5 hours a day watching TV.
>
>
>
> So I write Dave back a -very- long message, and at this point I'm very upset
> about this. I lay blame for the problems we have right now where it needs to be
> placed, squarely on his shoulders. He was the reason the white paper was given
> up on, as if it wasn't Dave's way, it was no good. Yeah, I'll admit my message
> to him was long, and rather blunt. I was direct with him, and factually telling
> him it's come to a point where change has to occur, or the network will change
> around him.
>
>
>
> Well, here was his response to that:
>
>
>
>
>
>
>
>
>
>
>
> Dave,
>
>
>
> In the time your spent wiring this email, you would could have instead been
> making progress on the a white paper your said you would complete last year.
> Don't blame the client developers for something you knew that was needed over a
> year ago.
>
>
>
> Dave Helms
>
>
>
>
>
>
>
>
>
> Accusing me of not finishing a document that he opposed?!!?!?!?!?!! Excuse
> me? Draft 3 was what would have gotten published had he not objected to the
> hostname. The rest -is- done right now.
>
>
>
> Well if the NWS employee that is supposed to be our liaison to the developers
> is taking this attitude towards me, with the thousands I have spent to support
> CWOP, I hereby pull my official support from the program.
>
>
>
> I realize this will be a heavy hit for CWOP, as I take a full one half of the
> server capacity with me. I am greatly sorry for that. The last people I
> wanted to hurt were the ones I was out to help, but with Dave Helms at the
> "Helm", it's over for me. I cannot continue to be badgered, beaten, ignored,
> then finally blamed for it to begin with over a volunteer operation.
>
>
>
> If any of you would like to talk about this more, I encourage you to send me a
> private e-mail at dave@aprsfl.net. I'm sure this message will get me banned on
> this forum, so I'll not be checking back.
>
>
>
> I'm glad I've been able to help up till now, and I really am truly sorry for
> going out this way, but I just can't go on with this type of pervasive
> attitude.. Let alone one that is so callous towards the folks that this network
> was built for, ham radio operators.
>
>
>
> Regards,
>
> Dave Anderson
>
> KG4YZY