As many of you noticed, AnandTech has spent several hours offline today. We are still in recovery mode at the moment (as I write this, the site has been restored to a copy from November 25th), but now that our major restoration efforts are completed, I wanted to offer you guys a brief update on the status of AnandTech.
At around 13:00 UTC (5am PT) today, the on-site cloud storage for AnandTech’s hosting provider became corrupted. As a result, AnandTech (and some other sites) were brought offline. Due to the nature of the corruption and the need to begin restoration efforts ASAP, we opted to restore the site from an off-site cold storage backup, rather than trusting the questionable on-site storage.
This is the first time we’ve ever had to execute our off-site data recovery plan before. And while it meant AT took a bit longer to restore than would be ideal, ultimately everything worked out and proved the necessity for off-site backups.
We’re still working to restore content from the last few days. Articles will be back, but we’ve likely lost any comments and user account registrations/updates made since midday Friday. Sorry about that! And thank you for bearing with us during today's outage.
Facebook
Twitter
RSS
31 Comments
View All Comments
linuxgeex - Sunday, December 4, 2022 - link
Properly, it would mean an on-prem AWS-compatible storage engine or the like. But instead they're probably talking about taking snapshots of VMs, and learned the hard way that such snapshots corrupt databases badly.dontlistentome - Friday, December 2, 2022 - link
I remember when my colleague took out the company's website and community by deleting all the content (shift-delete is dangerous!).Our desks were right outside the CEO's office. That was a fun 30 minutes whilst we restored the backup from (luckily) 3 hours earlier that morning whilst making excuses about an issue with the hosting. We lost a little bit of content, but yeah, backups.
Do them, and don't assume cloud providers are doing it or testing them for you.
paulwatsonjr@aim.com - Friday, December 2, 2022 - link
reminds me of the old saying: Untested backups are nothing more than a false sense of security...schizoide - Friday, December 2, 2022 - link
I would find a bit more detail to be interesting.Was this logical or physical corruption? If physical did you have a DR site? If not, that's the obvious next step.
Actually if it was logical, I setup DR sites replicated with a delay of several hours, so that works too.
iq100 - Friday, December 2, 2022 - link
A want Anandtech to provide a detailed analysis of how they had a data losing OUTAGE.What was the actual hardware that failed.
What was the actual software that failed.
After providing the analysis, Anandtech should put on their technical hat and offer a hardware/software design that cannot suffer a data loss.
A design that that has no single point of failure.
Or state there is no such design.
I understand that this may take some time.
BUT PLEASE POST NOW/IMMEDIATELY THAT ANANDTECH WILL DO THIS, AND AN ESTIMATE DATE FOR PROVIDING SUCH ANALYSIS. Thanks!
GeoffreyA - Friday, December 2, 2022 - link
"A design that that has no single point of failure.Or state there is no such design."
I think it's fair to say the latter wins the prize.
Threska - Friday, December 2, 2022 - link
Only nation-states could afford it.iq100 - Friday, December 2, 2022 - link
oops .. duplicated some text, and NO WAY ON ANANDTECH TO EDIT (with history of edits), or even delete and create new post. Another Anandtech design issue?GeoffreyA - Saturday, December 3, 2022 - link
In the future, conceivably, there'll be storage off-Earth. Because, come to think of it, if Earth were wiped out, all our data would be lost, including our art, classical buildings, etc.ballsystemlord - Sunday, December 4, 2022 - link
That's right! An alien species might be interested in us after we are gone. We need to leave something for them. ;)