Hetzner Cloud – Data loss incident
by bratao on Hacker News.
Data loss incident (snapshots) Dear Customer,
Unfortunately, we have to inform you that there was a data loss incident that affects a small amount of your snapshots on Hetzner Cloud.
All snapshots you create are stored on our highly available storage systems. The snapshot contents are distributed over multiple internal servers and data is stored in a way that allows up to two separate disks to fail without impacting data integrity.
This means the snapshot can still be accessed, even if two disks fail at the same time.
Due to a recent, very unfortunate series of events in one of our clusters, multiple disks failed in short succession and caused a small number of snapshots to become unavailable.
We immediately tried to recover the affected snapshots but unfortunately the data is lost and we have exhausted all our options. Affected snapshots in your account:
XXXXXXXXX The snapshots have been removed from our system as they are no longer accessible.
We sincerely hope this doesn’t cause too much trouble for you; we know losing data is the worst-case scenario. Also, we have added 20€ as Cloud Credits to your account (valid for one year). While we know that this will not bring back your data, we still hope that you will accept the gesture.
In response to this we will re-evaluate our snapshot cluster data replication strategies as well as our strategies for replacing disks and rebuilding redundancy after replacement. Best Regards,
Hetzner Cloud