Sunday, June 27, 2010

More Downtime for BWD Servers

The BWD servers are again offline, and this time a cause has been discovered. This is the second time this week, and third time in the last couple weeks. Very sorry for the inconvenience. It is likely no BWD accounts that are the cause of the problem, but unfortunately, we have been the victims of it.

Here is the message re: the downtime from the hosting company:

Causes / Solutions of Recent Downtime (BWD Server)
The BWD server has been down or unreachable 1 to 2 hours per day. There are two primary causes for this.

  1. Continuous brute-force attacks
  2. Frequent customer-inititated backups outside the normal automated backups.

The brute force attacks are very sophisticated and are coming from multiple locations around the world, most likely computers which have been compromised. Each attempt creates a number of log entries, which literally filled up one of the partitions on the drive (the cause of yesterday's downtime). The data center has been tweaking the firewall above the server to thwart these attacks as much as possible. Meanwhile, Platinum Server Management is working on tweaking server security.

The customer-initiated backups are most likely a result of the downtime itself. When there is downtime, there is a tendency for everyone to start generating and downloading backups on a more frequent basis. This uses up a lot of cpu and overloads the server. In short, fears about a crash become a self-fulfilling prophecy.

I have no doubt that the issue will soon be resolved. In the mean time, thank you for your understanding and your patience as we work through these issues.

The servers are currently still down as of 4:55pm ET. Will continue to monitor and update here when all is back online.

UPDATE: The servers are back online as of 5:45pm ET.

No comments: