Author Topic: WXforum.net outage - 10-12 November, 2019 .. we're back now.  (Read 2688 times)

0 Members and 1 Guest are viewing this topic.

Offline saratogaWX

  • Administrator
  • Forecaster
  • *****
  • Posts: 9257
  • Saratoga, CA, USA Weather - free PHP scripts
    • Saratoga-Weather.org
WXforum.net outage - 10-12 November, 2019 .. we're back now.
« on: November 12, 2019, 10:13:25 AM »
It all started when the database server started rejecting requests due to 'maximum number of concurrent connections'.

Even after placing the forum in Maintenance mode, and removing the direct SQL parts (to starve off connections), it still wouldn't resolve on its own.  It took three calls to ionos tech support to have their Level 2/3 DB experts finally get the database operational again.

On 12-Nov at 6 AM PST, the database was back, but needed attention to work with SMF.  I ran a repair tables operation, then ran the forum's internal find/fix errors and optimize tables.

Now we're back online, and I think that only in-flight postings at the time of the failure may have been lost, but nothing else.
Total Members: 15706  •  Total Posts: 378825  •  Total Topics: 36673

At 07:15, the forum was taken out of maintenance mode and resumed normal operation.  Thanks for your patience during this episode.

Best regards,
Ken
Ken True/Saratoga, CA, USA main site: saratoga-weather.org
Davis VP1+ FARS, Blitzortung RED, GRLevel3, WD, WL, VWS, Cumulus, Meteobridge
Free weather PHP scripts/website templates - update notifications on Twitter saratogaWXPHP

Offline mcrossley

  • Forecaster
  • *****
  • Posts: 1132
    • Wilmslow Astro
Re: WXforum.net outage - 10-12 November, 2019 .. we're back now.
« Reply #1 on: November 12, 2019, 10:17:48 AM »
Well done Ken - we can all do without these hassles.
Mark

Offline OldAlaskaGuy

  • Alaska, The Last American Frontier
  • Forecaster
  • *****
  • Posts: 399
  • Living in the "Last American Frontier"
Re: WXforum.net outage - 10-12 November, 2019 .. we're back now.
« Reply #2 on: November 12, 2019, 10:42:03 AM »
Good work. [tup]

Online Otis

  • (aka Paul)
  • Forecaster
  • *****
  • Posts: 724
    • Lake Huron Weather
Re: WXforum.net outage - 10-12 November, 2019 .. we're back now.
« Reply #3 on: November 12, 2019, 10:43:47 AM »
As always thanks for your efforts Ken, like you needed one more thing to do right!
Good work.

CW3699

Offline PaulMy

  • Forecaster
  • *****
  • Posts: 5509
    • KomokaWeather
Re: WXforum.net outage - 10-12 November, 2019 .. we're back now.
« Reply #4 on: November 12, 2019, 10:51:35 AM »
Well done Ken, and great  to see we're all still here. 

Also thank you for having that offline message as I am sure that avoided a large panic by us members.


Sincerely appreciated,
Paul

Offline Jasiu

  • Forecaster
  • *****
  • Posts: 947
    • LexMAWeather
Re: WXforum.net outage - 10-12 November, 2019 .. we're back now.
« Reply #5 on: November 12, 2019, 11:00:08 AM »
Thank you Ken! Glad to see the forum back up and running!
https://lexmaweather.info
On Mastodon: @LexMAWeather@toot.community

Offline galfert

  • Global Moderator
  • Forecaster
  • *****
  • Posts: 6822
Re: WXforum.net outage - 10-12 November, 2019 .. we're back now.
« Reply #6 on: November 12, 2019, 11:15:38 AM »
Than you Ken. I'm sure it was frustrating dealing with Level 1 support as it always is. Bunch of waste of time that is till you can get to the people that can really help. I was starting to show withdrawal symptoms, I didn't know what to do. So relieved now. Maybe I need a gadget free detox vacation. Nah who am I kidding that doesn't work.  :grin:

The best part was keeping us informed via index.php and weather-watch. I really appreciated that. (well we private emailed too but I'm just speaking for all others who I'm sure did appreciate it).
Ecowitt GW1000 | Meteobridge on Raspberry Pi
WU: KFLWINTE111  |  PWSweather: KFLWINTE111
CWOP: FW3708  |  AWEKAS: 14814
Windy: pws-f075acbe
Weather Underground Issue Tracking
Tele-Pole

Offline saratogaWX

  • Administrator
  • Forecaster
  • *****
  • Posts: 9257
  • Saratoga, CA, USA Weather - free PHP scripts
    • Saratoga-Weather.org
Re: WXforum.net outage - 10-12 November, 2019 .. we're back now.
« Reply #7 on: November 12, 2019, 11:42:15 AM »
Thanks for your kind comments.

Yes, it's really frustrating to 'work' with only a L1 support person when they're trying to fit your problem symptoms to a play-book of responses and you have to slowly/carefully explain the detailed problem and who is likely to be able to solve it (the L2/L3 DB server admins).
Then, there's no ongoing feedback, just a nebulous 'it should be fixed in X hours, and if not call back'.  Grrrr.  Shared hosting is cheap, and so is the canned support.

I use a shared hosting ionos contract for the forum (which also hosts northamericanweather.net and a test website) so it is easily portable if need be.  My other ionos contract is for a fairly beefy VPS (CentOS7) where I run Saratoga-weather.org plus a bunch of the regional network sites.  That support is EXCELLENT as I have a dedicated consultant who is a Unix geek and we can resolve any issues that arise on just the one phone call.. but, you do get what you pay for.. it's about 3x the $ as the shared hosting.

I know how important WXForum.net is to our members and I'll continue to offer status on multiple venues if we have an outage.

Best regards,
Ken
Ken True/Saratoga, CA, USA main site: saratoga-weather.org
Davis VP1+ FARS, Blitzortung RED, GRLevel3, WD, WL, VWS, Cumulus, Meteobridge
Free weather PHP scripts/website templates - update notifications on Twitter saratogaWXPHP

Offline Mabcmb

  • My WU page
  • Forecaster
  • *****
  • Posts: 328
Re: WXforum.net outage - 10-12 November, 2019 .. we're back now.
« Reply #8 on: November 12, 2019, 11:54:32 AM »
Thank you very very much for taken you time and trouble to get the site back up and running !!! I was having some serious weather  withdrawals symptoms  waiting on the site to come back up  :shock: . Once again thanks  [tup] [tup] [tup] !!!

Offline azkiwi

  • Senior Contributor
  • ****
  • Posts: 160
    • Maricopa, Sonoran Desert, Arizona
Re: WXforum.net outage - 10-12 November, 2019 .. we're back now.
« Reply #9 on: November 12, 2019, 01:23:59 PM »
Thank you Ken!



(another) Ken

 

anything