Maintenance Mode - overloaded DB connections

WXForum.net is having an overload issue with mySQL .. we are working to resolve. Please try again later.

Error is "#1203 - User [redacted] already has more than 'max_user_connections' active connections"
Same issue as last November. Their DB server thinks the connections are active. They aren't. The DB server needs to be reset. Convincing them to do that is the (ongoing) problem. GRRR. According to their Help info, Connections should time-out after 60 seconds of inactivity. It's been over 10hours, and no time-out.

This page was updated Wednesday, 01-Jan-2020 11:41 AM Pacific time

• 25-Dec-2018 21:30 problem first noticed. Waiting until tomorrow to get regular staff on tech support after Christmas.
  
  This problem has the same symptom as the one in November, 2019 that was also caused by the DB server itself. Now to get their tech support to understand that their 3rd level DB support needs to restart the mySQL service instance to drop the alleged connections (as was done last time).
• 26-Dec-2019 09:00 ionos tech support contacted. After 1h18m call with first/second level, their recommended bypass to 'starve' connections was implemented. Nevermind that the SMF connections were disabled 11 hrs ago when the problem first occured.
• 26-Dec-2019 09:45 Now waiting to see if their bypass worked
• 26-Dec-2019 10:55 Still have the 'max_user_connections' issue with the DB despite having no connections active from wxforum.net site. GRRR... calling ionos support again.
  Sigh.. support tech 'Mark' out to lunch. have to call back at 11:30.
• 26-Dec-2019 11:45 Called Tech Support again. Now awaiting for 'Mark' at 2nd level to answer the phone.
• 26-Dec-2019 12:18 Still on hold. Sigh. Cancelled call after 41 minutes on hold with no response.
• 26-Dec-2019 12:36 Calling ionos again.
• 26-Dec-2019 12:44 Talking with 'Kevin' (My 'personal' consultant) about the issue which still exists.
• 26-Dec-2019 12:57 The 2nd level hosting support is referring the case to developers (3rd level). It may take 24 to 48 hrs to resolve. All they really have to do is hup the mySQL server instance to drop the alleged connections (the site is not using mySQL at all, and hasn't been for 12hrs).
  More news as is available. Continuing to check at least hourly for phpMyAdmin access to the database.
• 26-Dec-2019 14:18 Received email from ionos. "Your support request has been accepted, and we will contact you directly as soon as we have new information."
• 26-Dec-2019 16:18 Been trying at 30 minute intervals. Still no joy.
• 26-Dec-2019 17:44 Still not fixed by 3rd level support.
• 26-Dec-2019 22:31 Alas, not fixed. Will check in the morning.
• 27-Dec-2019 06:36 Still not fixed by 3rd level support.
• 27-Dec-2019 07:42 Still awaiting fix by 3rd level support.
• 27-Dec-2019 08:23 Nope. Not fixed yet.
• 27-Dec-2019 09:29 Noticed that Tapatalk (mobiquo) was still enabled based on log analysis. Disabled it temporarily until we can get the database back. Database still not accessable via phpMyAdmin (still reporting max_user_connections active issue). Awaiting 3rd level fix from ionos.
• 27-Dec-2019 09:43 Received a note from ionos support

  Dear Mr. True,
  Thank you very much for reporting your issue on 12/26/2019. Our experts are already looking into it.
  As soon as new information becomes available, we will let you know.

• 27-Dec-2019 11:23 They may be working on it, but not very hard. DB still unavailable. Sigh.
• 27-Dec-2019 12:12 Still not fixed by 3rd level support.
• 27-Dec-2019 13:36 Sigh, problem persists.
• Tests continued (note that times are CST below) with no joy
  
  Fri Dec 27 15:09:31 CST 2019
  mysqladmin: connect to server at 'db[redacted].db.1and1.com' failed
  error: 'User dbo[redacted] already has more than 'max_user_connections' active connections'
  ...
  Fri Dec 27 21:04:33 CST 2019
  mysqladmin: connect to server at 'db[redacted].db.1and1.com' failed
  error: 'User dbo[redacted] already has more than 'max_user_connections' active connections'
• 27-Dec-2019 19:11 I'll check again in the morning... sigh.
• 28-Dec-2019 07:47 Sigh. Ran cron to check at 10 minute intervals. All results still show same issue.
  error: 'User dbo[redacted] already has more than 'max_user_connections' active connections'
  Time to call ionos support again.
• 28-Dec-2019 08:10 Just finished call to ionos support. Trouble ticket is with 'developers'. They haven't responded. Escalated ticket again citing excessive downtime and very dissatisfied customer. They will escalate again. Meanwhile, we wait.
• 28-Dec-2019 09:10 Still running the cron every 10 minutes and monitoring. Same error persists.
• 28-Dec-2019 13:47 Same error persists.
• 29-Dec-2019 07:55 Morning has broken, and so is the DB access (still). Sigh.
• 29-Dec-2019 12:17 Time for the afternoon call to ionos support. Still broken. GRRRRR.
• 29-Dec-2019 13:11 Finished a long call with the supervisor on 2nd level support. No, they don't have access to 3rd level to escalate my ticket (BB-42886), but they can annotate the ticket with my extreme displeasure at the excessive delay in 3rd level working on the ticket. We tried a test to change the password for the DB user using the ionos control panel -- it blew up with a 500-Server Error (as expected). Now, I just noticed that in my ionos control panel, the DB status is now 'setting up'. Maybe progress is being made at last. The cron started reporting a different error "error: 'Access denied for user 'dbo[redacted]'@'[redacted server address]' (using password: [redacted])" at 13:00.
• 29-Dec-2019 14:34 Still showing as 'Setting Up' in ionos panel. Same error message as in above posting. Continuing to monitor every 10 minutes via cron.
• 29-Dec-2019 17:05 It's been 4 hours, still in 'Setting Up' status on ionos panel. Hopefully, they're doing something productive and the database will be recovered.
• 30-Dec-2019 08:41 Sigh, another morning and no joy. DB still in 'Setting Up' status on control panel. Maybe the day-shift of the experts at 3rd level will make progress today. Now 5 days of continuous outage.
• 30-Dec-2019 10:13 Received a note saying that issue BB-42886 was resolved. Replied to the email that it's not actually resolved yet as the DB is still marked as "Setting Up' in the ionos control panel so there's no phpMyAdmin access yet. Time to call support again.
• 30-Dec-2019 11:02 On the phone with ionos support contesting 'resolved' for ticket BIB-42886. Ionos control panel shows DB still in 'Setting up' status. They are now stating that the 'closed' message was not correct. The 3rd level is still working on the problem. Meanwhile, I continue checking via cron for access to the database.
• 30-Dec-2019 13:14 The main DB is still in Setting Up mode and not accessable. I did try a DB restore to a new database mySQL 5.7 from the latest backup I had on hand (backup taken 2019-12-17T1935 UTC) and it loaded ok. The last posting on that date was topic=38379.msg394967 by me "Re: HELP - Weather Widget Bombs Out with Haze and Mist by saratogaWX" dated December 17, 2019, 11:19:23 Pacific.
  I'm not putting that version online as it would mean losing all postings (etc) from that date to December 25, but I do have that solution in reserve if ionos is unsuccessful in restoring the original database.
• 30-Dec-2019 19:26 Ok, I take it back (a bit).. the 2019-12-17 version is now online, but in maintenance mode (so nobody but admins can access it). I'm leaving the index.html page up to document what is going on with ionos. As of this time, the original database is still 'Setting up' and not accessable.
  I've taken some filesystem backups to make sure when we get the original database back, my messing about with the older backup version can be undone if needed.
• 31-Dec-2019 07:48 Email arrived from ionos support at 12:33 AM this morning saying the problem database was back online. Sure enough, it now appears online in the control panel. I'm doing repairs and taking another backup and will put the forum back online shortly. Our long outage is finally coming to an end.
• 31-Dec-2019 10:28 The forum was put online again at 9:00 AM today. I thought a synopsis of what was done to restore service is in order:
  I had access to the recovered database via phpMyAdmin via the ionos control panel. I did a repair tables (and it crashed), did a repair tables on the messages data base and it worked. Then did a mysql cli backup dump. Created a NEW database on a separate server and reloaded the dump using mysql cli. Changed the forum to use the new database, then used SMF to find/fix any errors and optimize tables .. no errors found and only minor optimization. The database is now at 895MB.
  A side note.. I'd tried mySQL 5.7 with the older Dec 17th dump, and it balloned to 1230MB v.s. 882MB for mySQL 5.5 so I've stuck with mySQL 5.5 where the limit is 1024MB for a single database. Because we're close to that limit, I'm exploring converting to a VPS running the site+mySQL where I can control the whole thing -- I do intend to keep it separate from my existing VPS for Saratoga-weather.org so it would continue to be portable in the future.
  That's the story.. Happy New Year to all.
• 01-Jan-20 11:36 Received followup call from ionos support at 07:30 this morning (unfortunately, woke up the wife). They explained that the developers couldn't find the underlying problem so they moved the database to a different mySQL server to get it operational again. Funny thing, yesterday, I'd created a new DB on yet another server to use as the original server had experienced the problem twice (November and December 2019). So, net of the exercise: 3rd level was clueless about root cause and just bypassed the issue by moving the database to a new mySQL server address. Sigh. Really time for me to spin up a VPS and take over the full admin and not rely on the shared hosting environment support with ionos anymore.