WXforum.net

Administration => WXForum.net Announcements => Topic started by: saratogaWX on November 12, 2019, 10:13:25 AM

Title: WXforum.net outage - 10-12 November, 2019 .. we're back now.
Post by: saratogaWX on November 12, 2019, 10:13:25 AM
It all started when the database server started rejecting requests due to 'maximum number of concurrent connections'.

Even after placing the forum in Maintenance mode, and removing the direct SQL parts (to starve off connections), it still wouldn't resolve on its own.  It took three calls to ionos tech support to have their Level 2/3 DB experts finally get the database operational again.

On 12-Nov at 6 AM PST, the database was back, but needed attention to work with SMF.  I ran a repair tables operation, then ran the forum's internal find/fix errors and optimize tables.

Now we're back online, and I think that only in-flight postings at the time of the failure may have been lost, but nothing else.
Total Members: 15706  •  Total Posts: 378825  •  Total Topics: 36673

At 07:15, the forum was taken out of maintenance mode and resumed normal operation.  Thanks for your patience during this episode.

Best regards,
Ken
Title: Re: WXforum.net outage - 10-12 November, 2019 .. we're back now.
Post by: mcrossley on November 12, 2019, 10:17:48 AM
Well done Ken - we can all do without these hassles.
Title: Re: WXforum.net outage - 10-12 November, 2019 .. we're back now.
Post by: OldAlaskaGuy on November 12, 2019, 10:42:03 AM
Good work. [tup]
Title: Re: WXforum.net outage - 10-12 November, 2019 .. we're back now.
Post by: Otis on November 12, 2019, 10:43:47 AM
As always thanks for your efforts Ken, like you needed one more thing to do right!
Good work.
Title: Re: WXforum.net outage - 10-12 November, 2019 .. we're back now.
Post by: PaulMy on November 12, 2019, 10:51:35 AM
Well done Ken, and great  to see we're all still here. 

Also thank you for having that offline message as I am sure that avoided a large panic by us members.


Sincerely appreciated,
Paul
Title: Re: WXforum.net outage - 10-12 November, 2019 .. we're back now.
Post by: Jasiu on November 12, 2019, 11:00:08 AM
Thank you Ken! Glad to see the forum back up and running!
Title: Re: WXforum.net outage - 10-12 November, 2019 .. we're back now.
Post by: galfert on November 12, 2019, 11:15:38 AM
Than you Ken. I'm sure it was frustrating dealing with Level 1 support as it always is. Bunch of waste of time that is till you can get to the people that can really help. I was starting to show withdrawal symptoms, I didn't know what to do. So relieved now. Maybe I need a gadget free detox vacation. Nah who am I kidding that doesn't work.  :grin:

The best part was keeping us informed via index.php and weather-watch. I really appreciated that. (well we private emailed too but I'm just speaking for all others who I'm sure did appreciate it).
Title: Re: WXforum.net outage - 10-12 November, 2019 .. we're back now.
Post by: saratogaWX on November 12, 2019, 11:42:15 AM
Thanks for your kind comments.

Yes, it's really frustrating to 'work' with only a L1 support person when they're trying to fit your problem symptoms to a play-book of responses and you have to slowly/carefully explain the detailed problem and who is likely to be able to solve it (the L2/L3 DB server admins).
Then, there's no ongoing feedback, just a nebulous 'it should be fixed in X hours, and if not call back'.  Grrrr.  Shared hosting is cheap, and so is the canned support.

I use a shared hosting ionos contract for the forum (which also hosts northamericanweather.net and a test website) so it is easily portable if need be.  My other ionos contract is for a fairly beefy VPS (CentOS7) where I run Saratoga-weather.org plus a bunch of the regional network sites.  That support is EXCELLENT as I have a dedicated consultant who is a Unix geek and we can resolve any issues that arise on just the one phone call.. but, you do get what you pay for.. it's about 3x the $ as the shared hosting.

I know how important WXForum.net is to our members and I'll continue to offer status on multiple venues if we have an outage.

Best regards,
Ken
Title: Re: WXforum.net outage - 10-12 November, 2019 .. we're back now.
Post by: Mabcmb on November 12, 2019, 11:54:32 AM
Thank you very very much for taken you time and trouble to get the site back up and running !!! I was having some serious weather  withdrawals symptoms  waiting on the site to come back up  :shock: . Once again thanks  [tup] [tup] [tup] !!!
Title: Re: WXforum.net outage - 10-12 November, 2019 .. we're back now.
Post by: azkiwi on November 12, 2019, 01:23:59 PM
Thank you Ken!



(another) Ken