Author Topic: WXforum.net outage - 25 - 31 December, 2019 .. we're back now.  (Read 2170 times)

0 Members and 1 Guest are viewing this topic.

Offline saratogaWX

  • Administrator
  • Forecaster
  • *****
  • Posts: 8193
  • Saratoga, CA, USA Weather - free PHP scripts
    • Saratoga-Weather.org
WXforum.net outage - 25 - 31 December, 2019 .. we're back now.
« on: December 31, 2019, 11:35:01 AM »
It has been a long outage due to database server issues with Ionos/1and1 mySQL server.

As of 08:30am Pacific on 31-Dec-2019, I've put the forum back online after changing to a different mySQL server.  The last post based on the backup was:

Latest Post: "Re: Merry Christmas ever..." ( December 25, 2019, 15:05:44 )
https://www.wxforum.net/index.php?topic=38440.msg395390;boardseen#new

so we haven't lost any posts (AFAIK), just had the extended outage.  For those interested in the detailed log of the outage,
I've left it up here.  I've also modified the backup procedure to take daily DB dumps so we can quickly recover in the event this mySQL connection issue happens again.

Thanks for all your patience waiting for the forum restoration (and for not bombarding me with emails about the outage).

Have a Happy New Year to all.
Best regards,
Ken

ed.note: changed link to outage info to https://www.wxforum.net/outage-december-2019.html
« Last Edit: December 31, 2019, 01:41:12 PM by saratogaWX »
Ken True/Saratoga, CA, USA main site: saratoga-weather.org
Davis VP1+ FARS, Boltek-PCI/NexStorm, microSferics ToA, Blitzortung RED, GRLevel3, WD, WL, VWS, Cumulus, Meteobridge/hub
Free weather PHP scripts/website templates - update notifications on Twitter saratogaWXPHP

Offline racer3

  • Senior Member
  • **
  • Posts: 78
Re: WXforum.net outage - 25 - 31 December, 2019 .. we're back now.
« Reply #1 on: December 31, 2019, 11:37:44 AM »
Welcome back, and Happy New Year! [tup]

Offline Intheswamp

  • Early on one frosty morn'...
  • Forecaster
  • *****
  • Posts: 1467
  • Sure...I know enough to be dangerous, now.
    • Beeweather
Re: WXforum.net outage - 25 - 31 December, 2019 .. we're back now.
« Reply #2 on: December 31, 2019, 11:47:17 AM »
Been logging on to see what the status was.  Glad to see it back up and running.  Thanks for all the hard work you do, Ken.  It's good to be starting the new year off with WXForum.net up and running!!!!!  HAPPY NEWYEAR to you and yours and all of WXForum.net!!!!!!!!!!!!!!!!

Ed

Davis VP2, v3.12
Cumulus v1.9.4  Build 1079
Today's weather beats no weather at all!
www.beeweather.com
Nope, I still haven't replaced that humidity sensor...<grin>

Offline Mabcmb

  • My WU page
  • Forecaster
  • *****
  • Posts: 305
Re: WXforum.net outage - 25 - 31 December, 2019 .. we're back now.
« Reply #3 on: December 31, 2019, 11:58:12 AM »
Outstanding  [tup] [tup] [tup] [tup] [tup]

Offline PaulMy

  • Forecaster
  • *****
  • Posts: 4844
    • KomokaWeather
Re: WXforum.net outage - 25 - 31 December, 2019 .. we're back now.
« Reply #4 on: December 31, 2019, 12:08:56 PM »
Thanks Ken for getting things back up and running.  Your ongoing update blog certainly helped and gave confidence that things were being worked on.


I trust you were still able to enjoy some holiday spirit.


Back to enjoy,
Paul

Offline vreihen

  • El Niņo chaser
  • Forecaster
  • *****
  • Posts: 1105
  • K2BIG
Re: WXforum.net outage - 25 - 31 December, 2019 .. we're back now.
« Reply #5 on: December 31, 2019, 12:16:01 PM »
Is there a reason why you continue to host with Ionos/1and1 after these repeated demonstrations of their technical incompetence?  If you were hosting on Amazon AWS or a similar provider, you could have rebooted the MySQL instance and been back to the holiday celebration in minutes.  Just curious.....
WU Gold Stars for everyone! :lol:

Offline mikeym2m

  • Contributor
  • ***
  • Posts: 134
    • MikeyM's Thornhurst PA Weather
Re: WXforum.net outage - 25 - 31 December, 2019 .. we're back now.
« Reply #6 on: December 31, 2019, 12:27:23 PM »
Ken, thanks for all the Hard Work  [tup]

Happy New Year to you and your family!

 :grin:

Cheers

MikeyM

Offline galfert

  • Global Moderator
  • Forecaster
  • *****
  • Posts: 6272
Re: WXforum.net outage - 25 - 31 December, 2019 .. we're back now.
« Reply #7 on: December 31, 2019, 12:41:42 PM »
Ken,
Thank you for all the work in restoring the forum. Thank you for keeping the users informed via the index page, that was much appreciated. I can not believe how terrible Ionos handled this situation. It is not like it was the first time this has happened (since it happened last month). I sent a nice (cordial but stern) email to Ionos management expressing my thoughts on how badly this issue was being handled. I have a followup yet to send to them. I can not believe that I never got a response back from them since I sent it a couple of days ago. This situation exemplifies the importance of backups. Glad you had a almost recent backup (8 days) at hand but that you persisted on restoring everything. It was worth it.

Happy New Year!
« Last Edit: December 31, 2019, 01:41:57 PM by galfert »
Ecowitt GW1000 | Meteobridge on Raspberry Pi
WU: KFLWINTE111  |  PWSweather: KFLWINTE111
CWOP: FW3708  |  AWEKAS: 14814
Windy: pws-f075acbe
Weather Underground Issue Tracking
Tele-Pole

Offline saratogaWX

  • Administrator
  • Forecaster
  • *****
  • Posts: 8193
  • Saratoga, CA, USA Weather - free PHP scripts
    • Saratoga-Weather.org
Re: WXforum.net outage - 25 - 31 December, 2019 .. we're back now.
« Reply #8 on: December 31, 2019, 12:57:52 PM »
Thanks Ken for getting things back up and running.  Your ongoing update blog certainly helped and gave confidence that things were being worked on.


I trust you were still able to enjoy some holiday spirit.


Back to enjoy,
Paul
Hi Paul,
Yes, the Christmas was quite enjoyable and low key.  We had unexpected visits from Southern California of my neice and brother-in-law.  Christmas Eve celebration was with son and two granddaughters (ages 5 and 7).  Over the weekend, I did my 48th sing-it-yourself Messiah sing.
In all, very enjoyable, despite the monitoring/nagging of ionos progress about the database server outage.

Is there a reason why you continue to host with Ionos/1and1 after these repeated demonstrations of their technical incompetence?  If you were hosting on Amazon AWS or a similar provider, you could have rebooted the MySQL instance and been back to the holiday celebration in minutes.  Just curious.....
It's simple, really .. I've been with ionos/1and1 since 2004 starting with a shared server for Saratoga-weather.org.  In September, 2016, I changed Saratoga-weather.org from a Business Linux (shared) to a Virtual Server L Linux package and moved a bunch of the regional networks there also.  I upgraded to a Virtual Server Cloud/Linux/Plesk in February 2019 and that's where Saratoga-weather.org (and 9 regional networks) reside.

For WXForum.net, it was created in 2006 on Hostgator by capeweather(Chris) as a subsite to his capeweather.com site when weatherforum.net died and we resurrected it.  It stayed there until March, 2013 when it was moved to a new shared hosting at 1and1 and I took over admin of the service.  I wanted it to be separate from my Saratoga-weather.org site set so it could be somewhat easily moved if need be.  In the time since 2013, there have been only two major outages, both caused by 1and1/ionos mySQL servers complaining about max user connections for the DB user.

Since I'm footing the bill for hosting of WXForum.net, it just made sense to me to have it added on to my existing 1and1 contracts rather than setting up entirely new hosting with another hoster.  The devil I know and all that.

I am looking at rolling this shared hosting to a new VPS as my existing VPS has proven very stable, and the support for VPS is excellent.. calls about it are answered by knowledgeable linux geeks and problems have been fixed while on the phone call -- a stark difference between shared and dedicated hosting.
I also didn't want to have a usage-based VPS because the forum is subject to the vagaries of rogue bots and can wildly drive up usage whereas a dedicated VPS has a fixed monthly cost -- that's why AWS (etc) wasn't attractive to me at all.
Ken True/Saratoga, CA, USA main site: saratoga-weather.org
Davis VP1+ FARS, Boltek-PCI/NexStorm, microSferics ToA, Blitzortung RED, GRLevel3, WD, WL, VWS, Cumulus, Meteobridge/hub
Free weather PHP scripts/website templates - update notifications on Twitter saratogaWXPHP

Offline daneast

  • Member
  • *
  • Posts: 46
Re: WXforum.net outage - 25 - 31 December, 2019 .. we're back now.
« Reply #9 on: December 31, 2019, 01:19:52 PM »
Regarding hosting, I have been using RamNode for years and have been extremely happy with them. I'm operating a pretty large online service, that also has dedicated iOS and Android apps, and their hosting has been great.  $10 a month gets you a quadcore, SSD VPS instance with 4 TB of monthly bandwidth.  I very much doubt you would need more than that to run your forums (I also operated a large discussion forum several years back).  You can just run a MySQL server on the same instance as your apache server.  It looks like you're just a standard LAMP stack - so I would look for better hosting, and especially more direct control over your MySQL server.

Offline awsum140

  • Forecaster
  • *****
  • Posts: 351
Re: WXforum.net outage - 25 - 31 December, 2019 .. we're back now.
« Reply #10 on: December 31, 2019, 01:29:16 PM »
I tip my hat to you in appreciation of your patience!  I watched the blog up dates and thought about how I might have reacted, which would have been quite differently.  Have a great New Year and thanks again for keeping this running despite the "excellent" support you received.
If it's not a Boerboel it's just a dog.

KNJSOUTH86


Offline twcmaster

  • Senior Member
  • **
  • Posts: 94
Re: WXforum.net outage - 25 - 31 December, 2019 .. we're back now.
« Reply #11 on: December 31, 2019, 01:50:54 PM »
Thank you Ken for getting us back up and running, keeping us updated the entire time, and for all that you do!!!!!!

Offline SnowHiker

  • Forecaster
  • *****
  • Posts: 324
Re: WXforum.net outage - 25 - 31 December, 2019 .. we're back now.
« Reply #12 on: December 31, 2019, 02:07:19 PM »
I was worried about the timing.  How many people may have got their first weather station for Christmas and wanted to tell people about it, search for info, and ask questions?  The first weather forum they find is continually off-line for days.  Hopefully didn't lose many potential new members. 

Not me, the only weather related gift I got is a storm glass.  I'll figure out how to get it hooked up and reporting to the internet later.   :lol:


Offline galfert

  • Global Moderator
  • Forecaster
  • *****
  • Posts: 6272
Re: WXforum.net outage - 25 - 31 December, 2019 .. we're back now.
« Reply #14 on: December 31, 2019, 02:52:00 PM »
We've been away for so long that it just seems so weird to not have more new content to read.  :sad:

This is probably what time travel feels like. Time passes for you but nothing has happened for anyone else. No wait...It's the other way around  :?
« Last Edit: December 31, 2019, 02:57:04 PM by galfert »
Ecowitt GW1000 | Meteobridge on Raspberry Pi
WU: KFLWINTE111  |  PWSweather: KFLWINTE111
CWOP: FW3708  |  AWEKAS: 14814
Windy: pws-f075acbe
Weather Underground Issue Tracking
Tele-Pole

Offline chief-david

  • Educational Weather
  • Forecaster
  • *****
  • Posts: 2589
  • Space Academy for Educators
    • Benilde-St. Margaret's Weather
Re: WXforum.net outage - 25 - 31 December, 2019 .. we're back now.
« Reply #15 on: December 31, 2019, 04:16:37 PM »
Thanks for everything Ken.
I also found your updates to be funny.



You can't phase me-I teach Middle School.
It's not you-It's WU.

Offline miraculon

  • Sunrise Side Weather
  • Forecaster
  • *****
  • Posts: 3773
  • KE8DAF
    • Sunrise Side Weather in Rogers City MI USA
Re: WXforum.net outage - 25 - 31 December, 2019 .. we're back now.
« Reply #16 on: December 31, 2019, 04:34:36 PM »
It looked like a real nightmare. Thanks for persisting and getting WxForum back up and running.

The updates were helpful if nothing else that efforts were being made to get it back online.

Three "thumbs up", Ken. (and Happy New Year!)

 [tup] [tup] [tup]

Greg H.


Blitzortung Stations #706 and #1682
CoCoRaHS: MI-PI-1
CWOP: CW4114 and KE8DAF-13
WU: KMIROGER7
Amateur Radio Callsign: KE8DAF

Offline droiddk

  • Forecaster
  • *****
  • Posts: 329
Re: WXforum.net outage - 25 - 31 December, 2019 .. we're back now.
« Reply #17 on: December 31, 2019, 04:37:46 PM »
Good job Ken!!

Thanks for all the updates.

Regards
« Last Edit: December 31, 2019, 05:58:14 PM by droiddk »

Offline yamiacaveman

  • Forecaster
  • *****
  • Posts: 340
    • Penn Lake Weather
Re: WXforum.net outage - 25 - 31 December, 2019 .. we're back now.
« Reply #18 on: December 31, 2019, 04:43:12 PM »
Thanks!!!!!

Happy New Year!!!

Offline Weatheroger

  • Senior Contributor
  • ****
  • Posts: 274
Re: WXforum.net outage - 25 - 31 December, 2019 .. we're back now.
« Reply #19 on: December 31, 2019, 05:35:17 PM »
Ken, you remind me of a dog on a pants leg, just won't let go. Thank you for your effort.  =D&gt; :thumbsup:

Offline 2thepoint

  • Senior Member
  • **
  • Posts: 50
    • NJ WeatherCat
Re: WXforum.net outage - 25 - 31 December, 2019 .. we're back now.
« Reply #20 on: December 31, 2019, 07:31:53 PM »
Thank You for all you do!

I think I was starting to tremor!

Offline btrip

  • Member
  • *
  • Posts: 6
    • South West Weather
Re: WXforum.net outage - 25 - 31 December, 2019 .. we're back now.
« Reply #21 on: December 31, 2019, 07:32:20 PM »
Thanks Ken for all you do.

Happy New Year

BTrip
B.Trip
DavisVP2/ Serial Data Logger, 2011 Mac Mini Server
High Sierra 10.13.6 2.3ghz Quad-Core Intel Core i7 16GB Ram
Arizona

Offline vinesweather

  • Firefighter
  • Senior Contributor
  • ****
  • Posts: 202
  • Be patient...I'm a firefighter not a computer whiz
    • The Vines Weather
Re: WXforum.net outage - 25 - 31 December, 2019 .. we're back now.
« Reply #22 on: January 01, 2020, 03:50:47 AM »
Your patience with the host is outstanding.  Well done sir!

Offline Da9L

  • Contributor
  • ***
  • Posts: 123
Re: WXforum.net outage - 25 - 31 December, 2019 .. we're back now.
« Reply #23 on: January 01, 2020, 06:30:37 AM »
I was worried about the timing.  How many people may have got their first weather station for Christmas and wanted to tell people about it, search for info, and ask questions?  The first weather forum they find is continually off-line for days.  Hopefully didn't lose many potential new members. 

Not me, the only weather related gift I got is a storm glass.  I'll figure out how to get it hooked up and reporting to the internet later.   :lol:

This was the case for my father in law. He got a froggit weather station like i had, so i could help him a little bit, but i setup mine the 23rd December, so it was limited what i knew too. I have a lot of questions now  :lol:

Offline saratogaWX

  • Administrator
  • Forecaster
  • *****
  • Posts: 8193
  • Saratoga, CA, USA Weather - free PHP scripts
    • Saratoga-Weather.org
Re: WXforum.net outage - 25 - 31 December, 2019 .. we're back now.
« Reply #24 on: January 01, 2020, 02:47:09 PM »
Here's a coda to the incident (also documented in the outage log)

01-Jan-20 11:36 Received followup call from ionos support at 07:30 this morning (unfortunately, woke up the wife). They explained that the developers couldn't find the underlying problem so they moved the database to a different mySQL server to get it operational again. Funny thing, yesterday, I'd created a new DB on yet another server to use as the original server had experienced the problem twice (November and December 2019). So, net of the exercise: 3rd level was clueless about root cause and just bypassed the issue by moving the database to a new mySQL server address. Sigh. Really time for me to spin up a VPS and take over the full admin and not rely on the shared hosting environment support with ionos anymore.

BTW, at the request of several of you, I've added a PayPal donation page to the site (and News rotation) for anyone wishing to help defray the costs of hosting the forum.  It uses my saratoga-weather.org PayPal account (which is autodebited monthly by ionos for the hosting).  Please feel no pressure to donate.. I've been happily funding this for years, and plan on continuing to fund it for the foreseeable future.
Ken True/Saratoga, CA, USA main site: saratoga-weather.org
Davis VP1+ FARS, Boltek-PCI/NexStorm, microSferics ToA, Blitzortung RED, GRLevel3, WD, WL, VWS, Cumulus, Meteobridge/hub
Free weather PHP scripts/website templates - update notifications on Twitter saratogaWXPHP