Endercloud - Notice history

Should be fixed after 30 minutes.

Discord: US10 is back up. This incident was automatically resolved by Instatus monitoring.

After extensive follow-ups and continued efforts, Leapswitch has finally provided us with disk-level data access to the affected systems. Immediately upon receiving access, we assembled a dedicated recovery team to analyze and recover any potentially corrupted data.

After nearly 6 continuous hours of intensive recovery and verification work, we are happy to confirm that all data has been successfully recovered and no corrupted data remains. The three servers that were previously reported as having lost data have now had their full data completely restored.

We can now officially confirm that this incident resulted in zero permanent data loss, and Endercloud continues to maintain a 0-incident record involving data loss.

We sincerely apologize for the inconvenience and uncertainty this situation caused. As already communicated, we have taken strong preventive measures to ensure this does not happen again, including permanently discontinuing Leapswitch and migrating our infrastructure to OVH for significantly improved reliability, protection, and recovery capabilities.

We truly appreciate your patience, trust, and continued support throughout this incident.

Thank you for standing with us.

Leapswitch has confirmed that data recovery attempts were unsuccessful and that, due to internal policies, they are unable to provide disk-level or raw data access for independent recovery.

As a result, data belonging to three servers on node IN04, which did not have panel-level backups created, was permanently lost. Full refunds and additional server compensation were provided to the affected users.

Node IN04 has been permanently discontinued, and all services are now operating on IN05, backed by OVH. OVH is a globally established provider used by major international hosting companies such as Shockbyte, BisectHosting, and SparkedHost, and offers significantly stronger DDoS mitigation and recovery capabilities.

We deeply regret this incident and sincerely apologize for the impact it caused. We have taken decisive steps to ensure this situation does not occur again.

Leapswitch informed us that they were continuing attempts to recover data from the affected disks. We continued to follow up for updates while assisting customers with redeployment and stabilization on the new IN05 (OVH-backed) infrastructure.

We did not receive further progress updates from Leapswitch due to limited operations during the holiday period.

During this time, we provisioned new infrastructure on OVH and began restoring services on the new IN05 node. Monitoring for IN04 is being permanently paused, as we are discontinuing this node following the incident.

Most users’ data was successfully restored using panel-level backups. These backups are created by users from the panel and are securely stored on external storage maintained by us.

At this stage, three servers were identified as having no panel-level backups, and their data status remained dependent on recovery efforts from the upstream provider.

To compensate for the downtime, affected users were offered renewal extensions for a duration of their choice.

Network connectivity was restored earlier today. However, during the server boot process, critical disk and filesystem errors were detected across the affected systems. Our upstream has indicated that these issues may have occurred due to the DDoS attack.

Given the severity of the situation and the risk posed to stored data, we immediately requested full data recovery and also requested raw disk access so we could attempt independent recovery.

Due to delayed responses, limited cooperation, and serious concerns regarding the provider’s ability to safely handle incidents of this nature, we made the decision to permanently discontinue Leapswitch as our upstream provider for our services and transition to OVH.

We have identified the cause of the outage as a sustained DDoS attack on our network over the past few days, which led our upstream providers to temporarily suspend network connectivity to protect their infrastructure. They have requested additional network and security implementations from our side to prevent a recurrence, and we have already submitted all required details and mitigation plans for their review. At this time, we do not have an estimated timeframe for restoration, as services will be resumed once the provider completes and approves their review. We sincerely apologize for the disruption caused; compensation will be provided after full service restoration. Once connectivity is restored, the network will operate under enhanced, provider-reviewed protections to significantly reduce the risk of this happening again.

We are currently experiencing a service disruption affecting some of our servers due to a network issue from one of our upstream network providers. Our team is actively investigating the issue and working closely with the provider to identify the root cause and restore connectivity as soon as possible. At this time, the affected services may be unreachable. We sincerely apologize for the inconvenience caused and truly appreciate your patience and understanding while we work to resolve this issue. Further updates will be shared as more information becomes available.