Author Topic: Saratoga-Weather.org goes VPS...  (Read 1539 times)

0 Members and 1 Guest are viewing this topic.

Offline saratogaWX

  • Administrator
  • Forecaster
  • *****
  • Posts: 9257
  • Saratoga, CA, USA Weather - free PHP scripts
    • Saratoga-Weather.org
Saratoga-Weather.org goes VPS...
« on: September 21, 2016, 05:55:40 PM »
It's been an active Summer'16 with reroofing and reconstructing the weather station (described here), but that wasn't all the fun.. I was kinda forced into a change in website hosting that allowed me to have all the stuff on the website work again.  Here's the background:

I've been hosting on shared Linux servers with 1and1 since 2006 -- first 'Business Linux' then when that package discontinued, with 'Dual Unlimited' hosting for $9 to $12/month.  When I first signed up, I was allowed free registration for 3 domains, so naturally I picked 'saratoga-weather.org' as one of them.  That became the primary domain for the website, and the main email domain (with forwards from all the other registered domains).

As time went on, various folks who had hosted Regional Weather networks had decided to no longer host/administer those networks and asked if I could take over hosting/admin for the networks.. sure, I said, and added them as websites to my hosting at 1and1.  No problem at all.  Because of that, I now host 4 Canada and 6 USA networks on the same site as Saratoga-weather.org.   All these sites depend on cron to routinely collect conditions from member stations, and my main site uses cron for a variety of background functions (Blitzortung maps, GR3 animation, sparkline graphs, etc) and some remote loading for some international regional networks (who didn't have cron availability).

Earlier this year, 1and1 decided to cap the number of concurrent FTP/SFTP connections per account to 5, so I routinely saw late/stale info on the website due to login failures.  Grrrr... A bit of re-architecting, using Fling and Bitvise SFTP bridge brought the site back into mostly full operation, but with the annoying problem of 'no login' for Dreamweaver website maintenance unless it was timed just right in the 5-minute update cycle.  This applied to updates to the 10 regional networks too, since they share my same 'account'.  Ok, livable but very annoying.

Then came August 1, 2016 when after 3:15pm, cron stopped processing jobs on the website's server.  Normally, a simple restart of the cron daemon would suffice to get it running, but that is something only the 3rd level system admins can do at 1and1.  I opened a ticket, and asked it be escalated to the server admins to fix.  That ticket was escalated to the domain admins (who have nothing to do with the problem), rejected and closed.  The next day, when I called to ask the status, they told me about the reject from the domain admins.  I said, of course.. now please route it to the server admins, which they said they would.  Unfortunately, it was late Friday afternoon, Pacific time, and I later found out that the server admins are all in Germany or Spain, and work only 9-5pm Central Europe time, Monday-Friday.  GRRRR.   About 2:30am on Aug 5, the cron process came alive again (I received a mail-message a minute as that's what I'd replaced the crontab with so I'd know exactly when it came alive).  By 7:00am Aug 5, I'd restored the crontab and the saratoga-weather.org site and 10 regional sites started collecting conditions again.   Shouting and getting angry with 1st/second level 1and1 doesn't help (it's cathartic but no more effective than the controlled anger I used with them). 

Given the cron issue and the FTP annoyance, I decided to switch to a Virtual Private Server where I could be server admin and restart stuff as needed.  I added a 'Linux M' VPS to my 1and1 account on August 4 and began configuring the new VPS.  It uses Plesk Virtuozzo for the VM configuration and Plesk Odin 12.0 for the software configuration.  I selected CentOS 6.8/Plesk 12.0.18 for the VM, and began setups for saratoga-weather.org and the 10 regional networks.  One issue immediately cropped up in testing, the PHP 5.6.25 had developed FMEM exceptions, so some pages would 500 out.  I had to backtrack to using 5.4.45 PHP to have the pages work without the 500 errors.

From August 4 through August 14, I moved each regional network, one at a time, from the old to the new VPS hosting.  This involved full setup on the VPS, then using the 1and1 domain control panel to transfer hosting from one contract to the VPS contract, then wait for the DNS change to propagate and make sure everything was working.  The next day, I'd shut off the cron collection for that network on the old hosting.  So it went for 10 networks, and all was well.

Now for the thorny bits... migrating saratoga-weather.org itself was more complicated because the domain was classed as 'included' in the original hosting contract, so I couldn't just transfer it to the new hosting contract.. first it had to be set 'free' from that contract.  Meanwhile, 'saratoga-weather.net' was available to serve the site on the VPS to bridge availability for the main website when the transfer was in progress.

August 15, I  'smart upgraded' the original contract hosting to 'Linux Unlimited' using the 1and1 panel, and this had the side effect of freeing up the three domains for transfer (and for me to pay for them separately).  That worked seamlessly.  I transferred two of the 3 'included' domains to the VPS contract and let DNS settle for a while.  I had to plan for how to handle email (which was on 1and1 IMAP servers) as that would be erased once I transfered saratoga-weather.org DNS.  I have been using Thunderbird for many years, and it stores locally the mbox it has from IMAP access to my main mail address, so I felt I was covered.

I put a 301 header in my main site Settings.php so every page would go to saratoga-weather.net temporarily (which was working on the VPS) and used the 1and1 panel to transfer saratoga-weather.org to the VPS contract.  With that, I lost the email connection (but had the mbox files), and waited for the DNS to settle.  As of August 15, 2016 Saratoga-weather.org is running on the VPS.

I did have to write a bash script (run by cron) to consolidate the access/error/ftp logs to a naming convention as was used on the shared hosting so my analysis scripts could still work.  I've been watching error_log and fixing missing stuff from the 15th through Sept 1st, and now what is missing is deliberate :)

Using

tar -czvf <filename>.tar.gz <directory>/   

to grab a copy of the files on the old site and

wget "..../<filename>.tar.gz"
tar -xvf <filename>.tar.gz

on the new site allowed easy move of the data from old to new site.

I have also expanded the VPS hosting from the "Linux M" (1CPU, 1.2GB/RAM, 50GB/disk) to the "Linux L" package (2CPU, 2GB/RAM, 50GB/disk) and seem to have plenty of headroom for growth.

The email has been a somewhat different story.. after 12 years, the mbox for INBOX had grown to 2.7GB (44268 messages) and Thunderbird refused to re-import it using the Import/Export mbox tools.  A handy Perl script (mboxsplit.pl) sliced up the mbox into separate files, and I could take the last several thousand or so and reimport to Thunderbird as an offline-folder.
I'm still working on getting back some emails, so if I haven't responded, please resend your query :)

In all, my monthly goes from $14 to $20/month and I have a lot more control over the server.  I did find that 1st level server support are actually quite useful and knowledgeable and can actually solve problems while you are on the phone.  Shared hosting 1st/second level support remains helpful only if your problem is a scripted response issue.  You are doomed to wait at least one business day (European time) if your shared host requires a server admin to fix.

The journey continues...

Best regards,
Ken
« Last Edit: September 21, 2016, 06:02:05 PM by saratogaWX »
Ken True/Saratoga, CA, USA main site: saratoga-weather.org
Davis VP1+ FARS, Blitzortung RED, GRLevel3, WD, WL, VWS, Cumulus, Meteobridge
Free weather PHP scripts/website templates - update notifications on Twitter saratogaWXPHP

Offline Jáchym

  • Meteotemplate Developer
  • Forecaster
  • *****
  • Posts: 8605
    • Meteotemplate
Re: Saratoga-Weather.org goes VPS...
« Reply #1 on: September 21, 2016, 06:02:37 PM »
LOL, looks like you were/are not bored :D

Btw. very good station placement, and I must admit I also love the surroundings.. the palm trees etc :)

Offline saratogaWX

  • Administrator
  • Forecaster
  • *****
  • Posts: 9257
  • Saratoga, CA, USA Weather - free PHP scripts
    • Saratoga-Weather.org
Re: Saratoga-Weather.org goes VPS...
« Reply #2 on: September 21, 2016, 06:06:45 PM »
Thanks Jáchym,

With the physical stuff (roofing, solar install, Davis refurbish/reconfigure) and the virtual stuff (hosting transition), it has been a non-boring Summer.

Note that the North looking photos show a stark difference in the lawn of the neighbor (ours too) as 5 years of CA drought and 30% reduction targets have left us much more 'brown' than 'green'.
Ken True/Saratoga, CA, USA main site: saratoga-weather.org
Davis VP1+ FARS, Blitzortung RED, GRLevel3, WD, WL, VWS, Cumulus, Meteobridge
Free weather PHP scripts/website templates - update notifications on Twitter saratogaWXPHP

Offline Jáchym

  • Meteotemplate Developer
  • Forecaster
  • *****
  • Posts: 8605
    • Meteotemplate
Re: Saratoga-Weather.org goes VPS...
« Reply #3 on: September 21, 2016, 06:22:23 PM »
Yes I see that, interesting.

Well I will never post photos of my station.... I feel ashamed about it, it is the very cheap FineOffset WH1080. Even though I created a DYI solar shield, it is useless. Simply because it is sited on an asphalt roof, 8m above the ground... unfortunately this really is the only place where I can put it, I do not have access to the garden and it is technically impossible to put anywhere else. The wind sensor is above the roof, but there are much higher buildings around me, so it is also very inaccurate. I take it the only purpose of my station is to feed data to the Demo of Meteotemplate and so the accuracy is not so important and I also stopped uploading data to CWOP etc. because I know it is misleading.
And I will not even go into my hardware and software set up... it is similiar thing, I use whatever is possible, but it is very limited, so I wont go into it and you should be very proud of yourself, your set up is exemplary and also shows how much time and effort you give into all this.