Author Topic: WXforum.net outage - 10-12 November, 2019 .. we're back now.  (Read 829 times)

0 Members and 1 Guest are viewing this topic.

Offline saratogaWX

  • Administrator
  • Forecaster
  • *****
  • Posts: 7530
  • Saratoga, CA, USA Weather - free PHP scripts
    • Saratoga-Weather.org
WXforum.net outage - 10-12 November, 2019 .. we're back now.
« on: November 12, 2019, 10:13:25 AM »
It all started when the database server started rejecting requests due to 'maximum number of concurrent connections'.

Even after placing the forum in Maintenance mode, and removing the direct SQL parts (to starve off connections), it still wouldn't resolve on its own.  It took three calls to ionos tech support to have their Level 2/3 DB experts finally get the database operational again.

On 12-Nov at 6 AM PST, the database was back, but needed attention to work with SMF.  I ran a repair tables operation, then ran the forum's internal find/fix errors and optimize tables.

Now we're back online, and I think that only in-flight postings at the time of the failure may have been lost, but nothing else.
Total Members: 15706    Total Posts: 378825    Total Topics: 36673

At 07:15, the forum was taken out of maintenance mode and resumed normal operation.  Thanks for your patience during this episode.

Best regards,
Ken
Ken True/Saratoga, CA, USA main site: saratoga-weather.org
Davis VP1+ FARS, Boltek-PCI/NexStorm, microSferics ToA, Blitzortung RED, GRLevel3, WD, WL, VWS, Cumulus, Meteobridge/hub
Free weather PHP scripts/website templates - update notifications on Twitter saratogaWXPHP

Offline mcrossley

  • Forecaster
  • *****
  • Posts: 457
    • Wilmslow Astro
Re: WXforum.net outage - 10-12 November, 2019 .. we're back now.
« Reply #1 on: November 12, 2019, 10:17:48 AM »
Well done Ken - we can all do without these hassles.
Mark

Offline Weatheroger

  • Senior Contributor
  • ****
  • Posts: 178
Re: WXforum.net outage - 10-12 November, 2019 .. we're back now.
« Reply #2 on: November 12, 2019, 10:42:03 AM »
Good work. [tup]
KCAGRASS150 WU page: https://www.wunderground.com/dashboard/pws/KCAGRASS150
Glenwood Station
Grass Valley Ca.
WS-1401-IP +  PM 2.5 Air Quality Sensor
Ambient page: https://dashboard.ambientweather.net/devices/public/cb606b61152ec0de5b4e6d97c5ab8af7
Ecowitt Page: https://www.ecowitt.net/home/index?id=6139

Online Otis

  • (aka Paul)
  • Forecaster
  • *****
  • Posts: 587
    • Lake Huron Weather
Re: WXforum.net outage - 10-12 November, 2019 .. we're back now.
« Reply #3 on: November 12, 2019, 10:43:47 AM »
As always thanks for your efforts Ken, like you needed one more thing to do right!
Good work.

Davis Vantage Pro2, Serial Data Logger,
WD 10.37S-(b60), VWS V15.00p03, Weatherlink 6.0.3,  Fling 2.35, Broadwave 2.0, Win 7 Pro 64Bit
KMICHEBO10 - CW3699 - Cheboygan, MI

Offline PaulMy

  • Forecaster
  • *****
  • Posts: 4552
    • KomokaWeather
Re: WXforum.net outage - 10-12 November, 2019 .. we're back now.
« Reply #4 on: November 12, 2019, 10:51:35 AM »
Well done Ken, and great  to see we're all still here. 

Also thank you for having that offline message as I am sure that avoided a large panic by us members.


Sincerely appreciated,
Paul
Davis Vantage Pro 2 Plus 24-FARS Wireless
Envoy/USB DataLogger/Cumulus v.1.9.4   -   Vue Console/WiFiLogger/CumulusMX
Komoka, ON  Canada
www.komokaweather.ca   www.komokaweather.com   www.komokaweather.com/weather28/  
www.komokaweather.com/pws/index.php
Blitzortung Station #1076 www.lightningmaps.org/blitzortung7/america/index.php?bo_page=statistics&bo_show=network&lang=en&bo_station_id=969


Offline Jasiu

  • Forecaster
  • *****
  • Posts: 594
    • LexMAWeather
Re: WXforum.net outage - 10-12 November, 2019 .. we're back now.
« Reply #5 on: November 12, 2019, 11:00:08 AM »
Thank you Ken! Glad to see the forum back up and running!

Online galfert

  • Global Moderator
  • Forecaster
  • *****
  • Posts: 3531
Re: WXforum.net outage - 10-12 November, 2019 .. we're back now.
« Reply #6 on: November 12, 2019, 11:15:38 AM »
Than you Ken. I'm sure it was frustrating dealing with Level 1 support as it always is. Bunch of waste of time that is till you can get to the people that can really help. I was starting to show withdrawal symptoms, I didn't know what to do. So relieved now. Maybe I need a gadget free detox vacation. Nah who am I kidding that doesn't work.  :grin:

The best part was keeping us informed via index.php and weather-watch. I really appreciated that. (well we private emailed too but I'm just speaking for all others who I'm sure did appreciate it).
WS-2000 & WS-2902A | Ecowitt GW1000 | WeatherBridge (Meteobridge)
WU: KFLWINTE111  |  PWSweather: KFLWINTE111
CWOP: FW3708  |  AWEKAS: 14814
Windy: pws-f075acbe
Weather Underground Issue Tracking
Tele-Pole

Offline saratogaWX

  • Administrator
  • Forecaster
  • *****
  • Posts: 7530
  • Saratoga, CA, USA Weather - free PHP scripts
    • Saratoga-Weather.org
Re: WXforum.net outage - 10-12 November, 2019 .. we're back now.
« Reply #7 on: November 12, 2019, 11:42:15 AM »
Thanks for your kind comments.

Yes, it's really frustrating to 'work' with only a L1 support person when they're trying to fit your problem symptoms to a play-book of responses and you have to slowly/carefully explain the detailed problem and who is likely to be able to solve it (the L2/L3 DB server admins).
Then, there's no ongoing feedback, just a nebulous 'it should be fixed in X hours, and if not call back'.  Grrrr.  Shared hosting is cheap, and so is the canned support.

I use a shared hosting ionos contract for the forum (which also hosts northamericanweather.net and a test website) so it is easily portable if need be.  My other ionos contract is for a fairly beefy VPS (CentOS7) where I run Saratoga-weather.org plus a bunch of the regional network sites.  That support is EXCELLENT as I have a dedicated consultant who is a Unix geek and we can resolve any issues that arise on just the one phone call.. but, you do get what you pay for.. it's about 3x the $ as the shared hosting.

I know how important WXForum.net is to our members and I'll continue to offer status on multiple venues if we have an outage.

Best regards,
Ken
Ken True/Saratoga, CA, USA main site: saratoga-weather.org
Davis VP1+ FARS, Boltek-PCI/NexStorm, microSferics ToA, Blitzortung RED, GRLevel3, WD, WL, VWS, Cumulus, Meteobridge/hub
Free weather PHP scripts/website templates - update notifications on Twitter saratogaWXPHP

Offline Mabcmb

  • My WU page
  • Contributor
  • ***
  • Posts: 133
Re: WXforum.net outage - 10-12 November, 2019 .. we're back now.
« Reply #8 on: November 12, 2019, 11:54:32 AM »
Thank you very very much for taken you time and trouble to get the site back up and running !!! I was having some serious weather  withdrawals symptoms  waiting on the site to come back up  :shock: . Once again thanks  [tup] [tup] [tup] !!!

Online azkiwi

  • Senior Contributor
  • ****
  • Posts: 151
    • Maricopa, Sonoran Desert, Arizona
Re: WXforum.net outage - 10-12 November, 2019 .. we're back now.
« Reply #9 on: November 12, 2019, 01:23:59 PM »
Thank you Ken!



(another) Ken