Topic: WXforum.net outage - 25 - 31 December, 2019 .. we're back now.

[saratogaWX]

It has been a long outage due to database server issues with Ionos/1and1 mySQL server.

As of 08:30am Pacific on 31-Dec-2019, I've put the forum back online after changing to a different mySQL server.  The last post based on the backup was:

Latest Post: "Re: Merry Christmas ever..." ( December 25, 2019, 15:05:44 )
https://www.wxforum.net/index.php?topic=38440.msg395390;boardseen#new

so we haven't lost any posts (AFAIK), just had the extended outage.  For those interested in the detailed log of the outage,
I've left it up here.  I've also modified the backup procedure to take daily DB dumps so we can quickly recover in the event this mySQL connection issue happens again.

Thanks for all your patience waiting for the forum restoration (and for not bombarding me with emails about the outage).

Have a Happy New Year to all.
Best regards,
Ken

ed.note: changed link to outage info to https://www.wxforum.net/outage-december-2019.html

[racer3]

Welcome back, and Happy New Year!

[Intheswamp]

Been logging on to see what the status was.  Glad to see it back up and running.  Thanks for all the hard work you do, Ken.  It's good to be starting the new year off with WXForum.net up and running!!!!!  HAPPY NEWYEAR to you and yours and all of WXForum.net!!!!!!!!!!!!!!!!

Ed

[Mabcmb]

Outstanding 

[PaulMy]

Thanks Ken for getting things back up and running.  Your ongoing update blog certainly helped and gave confidence that things were being worked on.


I trust you were still able to enjoy some holiday spirit.


Back to enjoy,
Paul

[vreihen]

Is there a reason why you continue to host with Ionos/1and1 after these repeated demonstrations of their technical incompetence?  If you were hosting on Amazon AWS or a similar provider, you could have rebooted the MySQL instance and been back to the holiday celebration in minutes.  Just curious.....

[mikeym2m]

Ken, thanks for all the Hard Work 

Happy New Year to you and your family!

 

Cheers

MikeyM

[galfert]

Ken,
Thank you for all the work in restoring the forum. Thank you for keeping the users informed via the index page, that was much appreciated. I can not believe how terrible Ionos handled this situation. It is not like it was the first time this has happened (since it happened last month). I sent a nice (cordial but stern) email to Ionos management expressing my thoughts on how badly this issue was being handled. I have a followup yet to send to them. I can not believe that I never got a response back from them since I sent it a couple of days ago. This situation exemplifies the importance of backups. Glad you had a almost recent backup (8 days) at hand but that you persisted on restoring everything. It was worth it.

Happy New Year!

[saratogaWX]

Thanks Ken for getting things back up and running.  Your ongoing update blog certainly helped and gave confidence that things were being worked on.


I trust you were still able to enjoy some holiday spirit.


Back to enjoy,
Paul

Hi Paul,
Yes, the Christmas was quite enjoyable and low key.  We had unexpected visits from Southern California of my neice and brother-in-law.  Christmas Eve celebration was with son and two granddaughters (ages 5 and 7).  Over the weekend, I did my 48th sing-it-yourself Messiah sing.
In all, very enjoyable, despite the monitoring/nagging of ionos progress about the database server outage.

Is there a reason why you continue to host with Ionos/1and1 after these repeated demonstrations of their technical incompetence?  If you were hosting on Amazon AWS or a similar provider, you could have rebooted the MySQL instance and been back to the holiday celebration in minutes.  Just curious.....

It's simple, really .. I've been with ionos/1and1 since 2004 starting with a shared server for Saratoga-weather.org.  In September, 2016, I changed Saratoga-weather.org from a Business Linux (shared) to a Virtual Server L Linux package and moved a bunch of the regional networks there also.  I upgraded to a Virtual Server Cloud/Linux/Plesk in February 2019 and that's where Saratoga-weather.org (and 9 regional networks) reside.

For WXForum.net, it was created in 2006 on Hostgator by capeweather(Chris) as a subsite to his capeweather.com site when weatherforum.net died and we resurrected it.  It stayed there until March, 2013 when it was moved to a new shared hosting at 1and1 and I took over admin of the service.  I wanted it to be separate from my Saratoga-weather.org site set so it could be somewhat easily moved if need be.  In the time since 2013, there have been only two major outages, both caused by 1and1/ionos mySQL servers complaining about max user connections for the DB user.

Since I'm footing the bill for hosting of WXForum.net, it just made sense to me to have it added on to my existing 1and1 contracts rather than setting up entirely new hosting with another hoster.  The devil I know and all that.

I am looking at rolling this shared hosting to a new VPS as my existing VPS has proven very stable, and the support for VPS is excellent.. calls about it are answered by knowledgeable linux geeks and problems have been fixed while on the phone call -- a stark difference between shared and dedicated hosting.
I also didn't want to have a usage-based VPS because the forum is subject to the vagaries of rogue bots and can wildly drive up usage whereas a dedicated VPS has a fixed monthly cost -- that's why AWS (etc) wasn't attractive to me at all.

[daneast]

Regarding hosting, I have been using RamNode for years and have been extremely happy with them. I'm operating a pretty large online service, that also has dedicated iOS and Android apps, and their hosting has been great.  $10 a month gets you a quadcore, SSD VPS instance with 4 TB of monthly bandwidth.  I very much doubt you would need more than that to run your forums (I also operated a large discussion forum several years back).  You can just run a MySQL server on the same instance as your apache server.  It looks like you're just a standard LAMP stack - so I would look for better hosting, and especially more direct control over your MySQL server.

[awsum140]

I tip my hat to you in appreciation of your patience!  I watched the blog up dates and thought about how I might have reacted, which would have been quite differently.  Have a great New Year and thanks again for keeping this running despite the "excellent" support you received.

[twcmaster]

Thank you Ken for getting us back up and running, keeping us updated the entire time, and for all that you do!!!!!!

[SnowHiker]

I was worried about the timing.  How many people may have got their first weather station for Christmas and wanted to tell people about it, search for info, and ask questions?  The first weather forum they find is continually off-line for days.  Hopefully didn't lose many potential new members. 

Not me, the only weather related gift I got is a storm glass.  I'll figure out how to get it hooked up and reporting to the internet later.   

[DRoberts]

Thanks for your time and effort.

[galfert]

We've been away for so long that it just seems so weird to not have more new content to read. 

This is probably what time travel feels like. Time passes for you but nothing has happened for anyone else. No wait...It's the other way around 

[chief-david]

Thanks for everything Ken.
I also found your updates to be funny.

[miraculon]

It looked like a real nightmare. Thanks for persisting and getting WxForum back up and running.

The updates were helpful if nothing else that efforts were being made to get it back online.

Three "thumbs up", Ken. (and Happy New Year!)

 

Greg H.

[droiddk]

Good job Ken!!

Thanks for all the updates.

Regards

[yamiacaveman]

Thanks!!!!!

Happy New Year!!!

[OldAlaskaGuy]

Ken, you remind me of a dog on a pants leg, just won't let go. Thank you for your effort. 

[2thepoint]

Thank You for all you do!

I think I was starting to tremor!

[btrip]

Thanks Ken for all you do.

Happy New Year

BTrip

[vinesweather]

Your patience with the host is outstanding.  Well done sir!

[Da9L]

I was worried about the timing.  How many people may have got their first weather station for Christmas and wanted to tell people about it, search for info, and ask questions?  The first weather forum they find is continually off-line for days.  Hopefully didn't lose many potential new members. 

Not me, the only weather related gift I got is a storm glass.  I'll figure out how to get it hooked up and reporting to the internet later.   

This was the case for my father in law. He got a froggit weather station like i had, so i could help him a little bit, but i setup mine the 23rd December, so it was limited what i knew too. I have a lot of questions now 

[saratogaWX]

Here's a coda to the incident (also documented in the outage log)

01-Jan-20 11:36 Received followup call from ionos support at 07:30 this morning (unfortunately, woke up the wife). They explained that the developers couldn't find the underlying problem so they moved the database to a different mySQL server to get it operational again. Funny thing, yesterday, I'd created a new DB on yet another server to use as the original server had experienced the problem twice (November and December 2019). So, net of the exercise: 3rd level was clueless about root cause and just bypassed the issue by moving the database to a new mySQL server address. Sigh. Really time for me to spin up a VPS and take over the full admin and not rely on the shared hosting environment support with ionos anymore.

BTW, at the request of several of you, I've added a PayPal donation page to the site (and News rotation) for anyone wishing to help defray the costs of hosting the forum.  It uses my saratoga-weather.org PayPal account (which is autodebited monthly by ionos for the hosting).  Please feel no pressure to donate.. I've been happily funding this for years, and plan on continuing to fund it for the foreseeable future.