So here is what happened with the outage.
There were complaints regarding some users randomly being redirected to a site that tried to offload malware on them (myfilestore). From reading up on it this was related to a vulnerability in the vBulletin software that allowed there to be data injected into some SEO templates.
I logged in to check the settings, and decided to finally allow the Ubuntu 16.04 LTS update to happen. I foolishly didn't take a snapshot before hand, but DigitalOcean wants you to have the Virtual Machine shutdown when you do a snapshot, so I skipped it. dumb mistake.
And that's where the problems started. The site just wasn't coming back up, more frantic investigations and I found that PHPv5 had been uninstalled as it isn't supported by Ubunti 16.04 (took me a while because the error message was complaining about something else, not php)
I installed PHP7, and got the site back up, as I knew there were some issues relating to our version of vBulletin and PHP7, I did some small changes to allow the site to run, but in testing there were some issues with posting, so I turned the forum off (so the site was at least displaying an outage message). It was now 4:30am, and I had started at 8:30pm. So I went off for a few hours sleep.
I was up again at 9, with my beloved topping me up with copious amounts of coffee, and got stuck back into trying to fix things so that I could make posts (when the board is turned off, moderators and the like can still post). Sparko in the mean time had tried a few things, but they had been about as successful as my attempts.
As I was working through the day, with the odd break to make sure the kids were listening to their mother ...school summer holidays, we're ready for them to be over... I wasn't making much headway, I would fix something to make it compatible with php7, and then the next item would start giving errors. So when it got towards Saturday evening I decided that the best option was going to be to try and backup the database, restore to the previous VM backup, and then try and restore the database.
With that in mind I backed up the db and started a download of it to my pc. It's a 9GB file so I knew it would take a while and left it overnight. However we had a BIG storm on Saturday night, and my internet connection dropped out and the download didn't complete.
I didn't restart it on Sunday morning as I was on Sound at church and had to be there early.
Come Sunday afternoon and I got the download restarted, and it took 3 hours to complete(the internet connection was still not overly happy following the storm so was running slower), but I had the DB backed up off the VM and was then able to restore the backup from 2 days before the outage.
I then spoke to Sparko and said that while I could restore the DB, it was going to take up to 6 hours to upload to the server before I could restore it, and I really wanted the site back up and running. So we made an executive decision to not proceed with restoring the missing two days, and just to turn the forum back on, I then applied the suggested advice regarding redirects that started this whole thing.
Where do we go from here with it?
We're investigating our options around upgrading to vBulletin 5 as it supports both php5 and php7, we'll give more information on that later in the year as there is a licence cost involved, and people have been not overly happy with vBulletin5 although it seems to have improved in the last couple of years, so we will want to test it out in a Dev environment before we make any final decisions.
vBulletin is working on a version of v4 that does support php7 but it's still in it's early beta stages with no release date in sight.
Once we've done that we will look at running the update to Ubuntu 16.04.
There were complaints regarding some users randomly being redirected to a site that tried to offload malware on them (myfilestore). From reading up on it this was related to a vulnerability in the vBulletin software that allowed there to be data injected into some SEO templates.
I logged in to check the settings, and decided to finally allow the Ubuntu 16.04 LTS update to happen. I foolishly didn't take a snapshot before hand, but DigitalOcean wants you to have the Virtual Machine shutdown when you do a snapshot, so I skipped it. dumb mistake.
And that's where the problems started. The site just wasn't coming back up, more frantic investigations and I found that PHPv5 had been uninstalled as it isn't supported by Ubunti 16.04 (took me a while because the error message was complaining about something else, not php)
I installed PHP7, and got the site back up, as I knew there were some issues relating to our version of vBulletin and PHP7, I did some small changes to allow the site to run, but in testing there were some issues with posting, so I turned the forum off (so the site was at least displaying an outage message). It was now 4:30am, and I had started at 8:30pm. So I went off for a few hours sleep.
I was up again at 9, with my beloved topping me up with copious amounts of coffee, and got stuck back into trying to fix things so that I could make posts (when the board is turned off, moderators and the like can still post). Sparko in the mean time had tried a few things, but they had been about as successful as my attempts.
As I was working through the day, with the odd break to make sure the kids were listening to their mother ...school summer holidays, we're ready for them to be over... I wasn't making much headway, I would fix something to make it compatible with php7, and then the next item would start giving errors. So when it got towards Saturday evening I decided that the best option was going to be to try and backup the database, restore to the previous VM backup, and then try and restore the database.
With that in mind I backed up the db and started a download of it to my pc. It's a 9GB file so I knew it would take a while and left it overnight. However we had a BIG storm on Saturday night, and my internet connection dropped out and the download didn't complete.
I didn't restart it on Sunday morning as I was on Sound at church and had to be there early.
Come Sunday afternoon and I got the download restarted, and it took 3 hours to complete(the internet connection was still not overly happy following the storm so was running slower), but I had the DB backed up off the VM and was then able to restore the backup from 2 days before the outage.
I then spoke to Sparko and said that while I could restore the DB, it was going to take up to 6 hours to upload to the server before I could restore it, and I really wanted the site back up and running. So we made an executive decision to not proceed with restoring the missing two days, and just to turn the forum back on, I then applied the suggested advice regarding redirects that started this whole thing.
Where do we go from here with it?
We're investigating our options around upgrading to vBulletin 5 as it supports both php5 and php7, we'll give more information on that later in the year as there is a licence cost involved, and people have been not overly happy with vBulletin5 although it seems to have improved in the last couple of years, so we will want to test it out in a Dev environment before we make any final decisions.
vBulletin is working on a version of v4 that does support php7 but it's still in it's early beta stages with no release date in sight.
Once we've done that we will look at running the update to Ubuntu 16.04.
Comment