On 10 March 2021 there was a major fire at a large data centre in France and 3.6 million websites being hosted on OVH Cloud were taken offline. BWF does not use this facility but we have had many a conversation with OVH Staff at conferences such as Cloufest and they are good people. No OVH staff members were hurt or injured so that is a blessing. The fire completely destroyed one of their Strasbourg data centre buildings. The CEO of OVH tweeted to his customers instructing them to ‘Implement your Disaster Recovery plan’ as there was no hope of retrieving any data from the destroyed data centre.
There are many third party news articles about this fire but three are pasted below for your convenience.
https://news.netcraft.com/archives/2021/03/10/ovh-fire.html
https://datacenterfrontier.com/ovh-data-center-in-france-destroyed-by-fire-all-staff-safe/
https://www.reuters.com/article/us-france-ovh-fire-idUSKBN2B20NU
What if… this happened in the Data Centre BWF use?
This fire shows that even with the best efforts and fire supression systems it is possible for a professional data centre to be destroyed by fire. For all our Managed Clients we always take backups to a completely different data centre. Whilst one of the data centres we use being destroyed by fire wouild be a catastrophy we would still have backup data in another facility. Yes it would be a massive headache and inconvenience but with a little bit of time and a lot of team effort we could relatively quickly make the data available to our clients and work to restore client data to another facility.
It’s important to note though that our backups are always deemed to be best effort and we do check them manually on a monthly audit task for all servers we have. We’ve never had backups fail at the same time as a production server but it’s always a possibility. There is a chance our backups could fail and our terms of service state that although we take backups we cannot guarantee them so as part of your disaster recovery planning it may be prudent to think about a secondary backup for those clients who livelihoods rely on their backups. Any additional backup products provided by us are cost effective and relatively inexpensive.
What Enhanced Backups are available?
As a company we do offer competitively priced enhanced backups for all our managed server clients and we can tailor a backup plan to suit your exact needs. Some examples are shown below in the bullet points.
- We can provide a cheap but effective additional private backup server. Private Solutions start at £10 per month up to £99 per month for a fully dedicated server with 16TB storage space. It’s even more unlikely that our standard backups AND your additional backup server would not be working and helps ensure data is preserved in the event of a crisis.
- We can increase the number of restore points your server has for a small additional fee useful in providing more options should an account become compromised or should something not be noticed in time and our standard backups be over-written.
- For eCommerce clients we can provide MySQL Live Replication of your main database, useful if you need to ensure if something goes wrong you lose no client orders or client data. We can also provide this across 2 different data centres for peace of mind.
- For eCommerce clients we can provide hourly database backups and we can sync these to your existing backup server (which is always in a different data centre). This means you can be assured the most data you will lose in the event of a catastrophie will be 60 minutes. Most eCommerce clients see this as acceptable as they could reconstruct an hours worth of orders from emails and payment gateway logs for instance. This is extremely cost effective and can be set up quickly.
- For shared hosting clients we encourage as with all data you deem to be important that you keep local backups on a regular basis. It’s really simple to backup from your control panel and our team can help you with this on request. Having a local backup means if the worst happens you have your data to restore. We can also sell you remote backup space and set up your shared hosting account to backup daily to that space automatically.
- For our managed server clients we can help you set up standard cPanel backups to store your data off site on a server you have with someone else (Amazon for instance). Many clients as part of due diligence feel this is an extra step to ensure their data is securely held independently of their hosting provider. Talk to our team if you need help setting this up. We may need to charge a small fee for staff time to set this up but we’re here to offer advice on this.
Do you have a Disaster Recovery Plan?
This complete incident highlights the need for every client whose livelihood depends on their website being on line to sit down and take steps to have their own Disaster Recovery Plan. Part of that plan should ask yourself if the backups we provide as part of your service are suitable for you or if you should look to consider extra backups or third party backups (independent from BWF) as part of your planning.
Our Management Team are available to talk to any client who wants advice on what additional steps they can take to ensure they are confident if the worst did happen they could recover in as fast a time as possible.
Finally we hope you never need a Disaster Recovery Plan but this fire has indicated that it is not something it would be wise to ignore. We want to ensure all our clients at least think through what they would do if our Data Centre was destroyed by fire (we sincerely hope it never happens of course).
