WXforum.net
Administration => WXForum.net Announcements => Topic started by: saratogaWX on December 31, 2019, 11:35:01 AM
-
It has been a long outage due to database server issues with Ionos/1and1 mySQL server.
As of 08:30am Pacific on 31-Dec-2019, I've put the forum back online after changing to a different mySQL server. The last post based on the backup was:
Latest Post: "Re: Merry Christmas ever..." ( December 25, 2019, 15:05:44 )
https://www.wxforum.net/index.php?topic=38440.msg395390;boardseen#new
so we haven't lost any posts (AFAIK), just had the extended outage. For those interested in the detailed log of the outage,
I've left it up here (https://www.wxforum.net/outage-december-2019.html). I've also modified the backup procedure to take daily DB dumps so we can quickly recover in the event this mySQL connection issue happens again.
Thanks for all your patience waiting for the forum restoration (and for not bombarding me with emails about the outage).
Have a Happy New Year to all.
Best regards,
Ken
ed.note: changed link to outage info to https://www.wxforum.net/outage-december-2019.html
-
Welcome back, and Happy New Year! [tup]
-
Been logging on to see what the status was. Glad to see it back up and running. Thanks for all the hard work you do, Ken. It's good to be starting the new year off with WXForum.net up and running!!!!! HAPPY NEWYEAR to you and yours and all of WXForum.net!!!!!!!!!!!!!!!!
Ed
-
Outstanding [tup] [tup] [tup] [tup] [tup]
-
Thanks Ken for getting things back up and running. Your ongoing update blog certainly helped and gave confidence that things were being worked on.
I trust you were still able to enjoy some holiday spirit.
Back to enjoy,
Paul
-
Is there a reason why you continue to host with Ionos/1and1 after these repeated demonstrations of their technical incompetence? If you were hosting on Amazon AWS or a similar provider, you could have rebooted the MySQL instance and been back to the holiday celebration in minutes. Just curious.....
-
Ken, thanks for all the Hard Work [tup]
Happy New Year to you and your family!
:grin:
Cheers
MikeyM
-
Ken,
Thank you for all the work in restoring the forum. Thank you for keeping the users informed via the index page, that was much appreciated. I can not believe how terrible Ionos handled this situation. It is not like it was the first time this has happened (since it happened last month). I sent a nice (cordial but stern) email to Ionos management expressing my thoughts on how badly this issue was being handled. I have a followup yet to send to them. I can not believe that I never got a response back from them since I sent it a couple of days ago. This situation exemplifies the importance of backups. Glad you had a almost recent backup (8 days) at hand but that you persisted on restoring everything. It was worth it.
Happy New Year!
-
Thanks Ken for getting things back up and running. Your ongoing update blog certainly helped and gave confidence that things were being worked on.
I trust you were still able to enjoy some holiday spirit.
Back to enjoy,
Paul
Hi Paul,
Yes, the Christmas was quite enjoyable and low key. We had unexpected visits from Southern California of my neice and brother-in-law. Christmas Eve celebration was with son and two granddaughters (ages 5 and 7). Over the weekend, I did my 48th sing-it-yourself Messiah sing.
In all, very enjoyable, despite the monitoring/nagging of ionos progress about the database server outage.
Is there a reason why you continue to host with Ionos/1and1 after these repeated demonstrations of their technical incompetence? If you were hosting on Amazon AWS or a similar provider, you could have rebooted the MySQL instance and been back to the holiday celebration in minutes. Just curious.....
It's simple, really .. I've been with ionos/1and1 since 2004 starting with a shared server for Saratoga-weather.org. In September, 2016, I changed Saratoga-weather.org from a Business Linux (shared) to a Virtual Server L Linux package and moved a bunch of the regional networks there also. I upgraded to a Virtual Server Cloud/Linux/Plesk in February 2019 and that's where Saratoga-weather.org (and 9 regional networks) reside.
For WXForum.net, it was created in 2006 on Hostgator by capeweather(Chris) as a subsite to his capeweather.com site when weatherforum.net died and we resurrected it. It stayed there until March, 2013 when it was moved to a new shared hosting at 1and1 and I took over admin of the service. I wanted it to be separate from my Saratoga-weather.org site set so it could be somewhat easily moved if need be. In the time since 2013, there have been only two major outages, both caused by 1and1/ionos mySQL servers complaining about max user connections for the DB user.
Since I'm footing the bill for hosting of WXForum.net, it just made sense to me to have it added on to my existing 1and1 contracts rather than setting up entirely new hosting with another hoster. The devil I know and all that.
I am looking at rolling this shared hosting to a new VPS as my existing VPS has proven very stable, and the support for VPS is excellent.. calls about it are answered by knowledgeable linux geeks and problems have been fixed while on the phone call -- a stark difference between shared and dedicated hosting.
I also didn't want to have a usage-based VPS because the forum is subject to the vagaries of rogue bots and can wildly drive up usage whereas a dedicated VPS has a fixed monthly cost -- that's why AWS (etc) wasn't attractive to me at all.
-
Regarding hosting, I have been using RamNode for years and have been extremely happy with them. I'm operating a pretty large online service, that also has dedicated iOS and Android apps, and their hosting has been great. $10 a month gets you a quadcore, SSD VPS instance with 4 TB of monthly bandwidth. I very much doubt you would need more than that to run your forums (I also operated a large discussion forum several years back). You can just run a MySQL server on the same instance as your apache server. It looks like you're just a standard LAMP stack - so I would look for better hosting, and especially more direct control over your MySQL server.
-
I tip my hat to you in appreciation of your patience! I watched the blog up dates and thought about how I might have reacted, which would have been quite differently. Have a great New Year and thanks again for keeping this running despite the "excellent" support you received.
-
Thank you Ken for getting us back up and running, keeping us updated the entire time, and for all that you do!!!!!!
-
I was worried about the timing. How many people may have got their first weather station for Christmas and wanted to tell people about it, search for info, and ask questions? The first weather forum they find is continually off-line for days. Hopefully didn't lose many potential new members.
Not me, the only weather related gift I got is a storm glass. I'll figure out how to get it hooked up and reporting to the internet later. :lol:
-
Thanks for your time and effort.
-
We've been away for so long that it just seems so weird to not have more new content to read. :sad:
This is probably what time travel feels like. Time passes for you but nothing has happened for anyone else. No wait...It's the other way around :?
-
Thanks for everything Ken.
I also found your updates to be funny.
-
It looked like a real nightmare. Thanks for persisting and getting WxForum back up and running.
The updates were helpful if nothing else that efforts were being made to get it back online.
Three "thumbs up", Ken. (and Happy New Year!)
[tup] [tup] [tup]
Greg H.
-
Good job Ken!!
Thanks for all the updates.
Regards
-
Thanks!!!!!
Happy New Year!!!
-
Ken, you remind me of a dog on a pants leg, just won't let go. Thank you for your effort. =D> :thumbsup:
-
Thank You for all you do!
I think I was starting to tremor!
-
Thanks Ken for all you do.
Happy New Year
BTrip
-
Your patience with the host is outstanding. Well done sir!
-
I was worried about the timing. How many people may have got their first weather station for Christmas and wanted to tell people about it, search for info, and ask questions? The first weather forum they find is continually off-line for days. Hopefully didn't lose many potential new members.
Not me, the only weather related gift I got is a storm glass. I'll figure out how to get it hooked up and reporting to the internet later. :lol:
This was the case for my father in law. He got a froggit weather station like i had, so i could help him a little bit, but i setup mine the 23rd December, so it was limited what i knew too. I have a lot of questions now :lol:
-
Here's a coda to the incident (also documented in the outage log (https://www.wxforum.net/outage-december-2019.html))
01-Jan-20 11:36 Received followup call from ionos support at 07:30 this morning (unfortunately, woke up the wife). They explained that the developers couldn't find the underlying problem so they moved the database to a different mySQL server to get it operational again. Funny thing, yesterday, I'd created a new DB on yet another server to use as the original server had experienced the problem twice (November and December 2019). So, net of the exercise: 3rd level was clueless about root cause and just bypassed the issue by moving the database to a new mySQL server address. Sigh. Really time for me to spin up a VPS and take over the full admin and not rely on the shared hosting environment support with ionos anymore.
BTW, at the request of several of you, I've added a PayPal donation page (https://www.wxforum.net/donate.php) to the site (and News rotation) for anyone wishing to help defray the costs of hosting the forum. It uses my saratoga-weather.org PayPal account (which is autodebited monthly by ionos for the hosting). Please feel no pressure to donate.. I've been happily funding this for years, and plan on continuing to fund it for the foreseeable future.
-
Happy to send a few shekles your way for the "cause". Thank you for all your effort. [tup]
-
Happy to send a few shekles your way for the "cause". Thank you for all your effort. [tup]
Anyone who wishes to chip in to help Ken defray the costs of all he does to us can click on the "Donate" button (takes you to PayPal) at https://saratoga-weather.org/wxtemplates/ (https://saratoga-weather.org/wxtemplates/)
-
I was just about to make the same suggestion to set up a donation link. I think this link should be right on the home page of the forum. We all benefit from this forum. We all get a lot of benefit from Ken's hard work (and time!) and he shouldn't have to also foot the bill by himself. I'm glad to not see this forum plastered with banner ads as many other Internet forums do.
Let's get donation link right on the home page. I think the donation link could have a nice quote below it. "banner ad free since 2006, thank you for your donation"
-
Ok, I've taken the Donation link out of the rotating News and added a link under the News in the header for the site.
THANKS to the folks who have already donated!
-
Anyone who wishes to chip in to help Ken defray the costs of all he does to us
That was obviously supposed to be "of all he does FOR us" but it's pretty funny as is so I won't edit it! :grin:
-
Anyone who wishes to chip in to help Ken defray the costs of all he does to us
That was obviously supposed to be "of all he does FOR us" but it's pretty funny as is so I won't edit it! :grin:
I think both FOR 8-) and TO :twisted: is appropriate .. thanks for the laughs. =D> :lol:
-
Donation made. Your scripts along with several others which I modifed some for Wordpress have been a Godsend. Thanks.
http://weatherwx.net (http://weatherwx.net)
-
Ken,
Echoing all before, thank you for all that you do and have done! I would strongly encourage you to move to a business hosting site such as ICDSoft to host WxForum.. I have used them for at least 10+ years with outstanding support (when needed). Their packages are extremely competitive and 24/7 support is unmatched....
Jim
-
Things have now settled after our long outage and the recovery. As soon as we came up, a daily backup of the database was implemented and has been running flawlessly -- the most we'd loose would be one day of data if the DB server crashes again. As I said in the outage log (https://www.wxforum.net/outage-december-2019.html), I moved the database to a newer mySQL 5.5 instance on a different ionos server .. we're still on shared hosting, so still vulnerable to the vagaries of 3rd level support during mySQL connection issues.
With the SMF 2.0.17 update, some of the plugins started throwing massive Notice/Deprecated errors in the error log (which temporarily overfilled the mySQL server size limit of 1024Mb. Some whacking through some of the plugins to suppress those errata was needed and overnight, the error log was barely populated. Whew.
I'm letting the forum just hum along (while minding the error log) for the next couple of weeks. Thanks to the extraordinary number of donations made in the last 3 days (THANKS to all who contributed), I will be preparing a Virtual Private Server instance to migrate the forum (and northamericanweather.net) to hosting that can be fully controlled (A CentOS7/Plesk 18 instance) that will run a mySQL server on the same virtual host.
I'll keep you informed about the progress of that via the Bulletin area.
Best regards,
Ken